Systems and methods for lesion analysis

ABSTRACT

A system for facilitating lesion analysis is configurable to identify a user profile associated with a user accessing the system. The user profile indicates a radiology specialty associated with the user. The system is also configurable to access a plurality of cross-sectional medical images associated with a particular patient and identify a subset of cross-sectional medical images from the plurality of cross-sectional medical images that correspond to the radiology specialty indicated by the user profile. The system is also configurable to present the subset of cross-sectional medical images to the user in navigable form.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalPatent Application No. 62/942,126, filed Nov. 30, 2020 and entitled“SYSTEMS AND METHODS FOR LESION ANALYSIS.” The foregoing application isincorporated herein by reference in its entirety.

BACKGROUND

Assessment of changes in tumor burden is an important feature fordefining tumor response in clinical practice and clinical trials. Bothtumor shrinkage and development of disease progression are importantendpoints in clinical practice and clinical trials as these oftendetermine objective response to treatment. In order to standardize tumorresponse assessment, various response criteria have been described,including Response Evaluation Criteria in Solid Tumors (RECIST) version1.0 or more commonly version 1.1, modified RECIST (mRECIST), WorldHealth Organization (WHO) Criteria, Choi Criteria, Vascular Tumor Burden(VTB) Criteria, Morphology Attenuation Size and Structure (MASS)Criteria, immune-related Response Criteria (irRC), immune-related RECIST(irRECIST), Cheson Criteria, Lugano Classification lymphoma responsecriteria, International Working Group consensus response evaluationcriteria in lymphoma (RECIL), Positron Emission Tomography ResponseCriteria in Solid Tumors (PERCIST), European Organization for Researchand Treatment of Cancer (EORTC) Response Criteria, Prostate CancerWorking Group 3 (PCWG3) criteria, Response Assessment in Neuro-Oncology(RANO) Criteria, immune RANO (iRANO), International Myeloma WorkingGroup (IMWG) consensus criteria, etc.

In order to assess objective response, an estimate of the overall tumorburden at baseline is needed and used as a comparator for subsequentmeasurements. Each tumor response criteria specifies parameters thatdefine a measurable lesion at baseline. For example, RECIST 1.1 definesa non-nodal lesion as measurable if it measures ≥1.0 cm in long axis atbaseline and defines a lymph node as measurable if it measures ≥1.5 cmin short axis at baseline. When one or more measurable lesions arepresent at baseline, each tumor response criteria specifies whichlesions should be considered as target lesions. Target lesions aretypically selected based on being the largest in size or mostmetabolically active but also should lend themselves to reproduciblerepeated measurements. Most tumor response criteria limit the number oftotal target lesions and limit the number of target lesions per organ.For example, RECIST 1.1. limits the total number of target lesions to 5and the total number of target lesions per organ to 2. Each tumorresponse criteria specifies how the target lesions should be measured.For example, RECIST 1.1 states that non-nodal lesions should be measuredin the longest dimension on axial cross-sectional images, while lymphnodes should be measured in short axis on axial cross-sectional images.The total tumor burden is then a mathematical calculation made from theindividual target lesions. For example, the sum of the diameters(longest for non-nodal lesions, short axis for nodal lesions) for alltarget lesions is calculated and reported as the baseline sum diametersper RECIST 1.1.

The baseline measurements are used as a reference to characterizeobjective tumor regression or progression in the measurable dimension ofthe disease. All other lesions (or sites of disease) are identified asnon-target lesions. The site of disease of all non-target lesions shouldbe recorded at baseline. At subsequent time points, measurement ofnon-target lesions is not required, and these lesions are typicallyfollowed and defined as ‘complete response’ (CR), ‘unequivocalprogressive disease’ (PD), ‘non-CR/non-PD’, or ‘not evaluable’ (NE).Alternatively, the non-target lesions could be qualitatively evaluated,such as ‘present’, ‘absent’, ‘larger’, or ‘smaller’.

While most tumor response criteria utilize measured changes in targetlesion length or size as a means of defining objective response, somecriteria (e.g., Lugano, PERCIST and EORTC Response Criteria) utilizemeasured changes in target lesions radiotracer activity as a means ofdefining objective response, and other criteria use a combination ofboth. Different tumor response criteria may utilize different metrics,mathematical calculations, or cut points to define objective response,and computer implemented methods that automate one or more processes ormethod acts and/or ensure user compliance with one or more criteria maybe used to reduce errors and improve efficiency in tumor responseassessment.

A critical component of any tumor response criteria is the choice oftarget lesions on the baseline exam. In clinical practice and clinicaltrials, the choice of target lesions is at the discretion of thephysician reviewer, which could be a radiologist, oncologist, radiationoncologist, surgeon, etc. Most tumor response criteria provide guidanceon target lesion selection. For example, RECIST 1.1 provides guidance onwhich lesions are measurable or non-measurable and then providesadditional details on how to select target lesions. In general targetlesions and lymph nodes are selected based on their size, though thetarget lesions must be representative of all involved organs and shouldlend themselves to reproducible repeated measurements. Furthermore,tracking of target lesions over time is advantageous for obtainingaccurate and precise objective response.

Conventional methods for tracking lesions (e.g., target lesions and/ornon-target lesions) include navigating to an appropriate cross-sectionalimage, identifying a lesion for analysis, and recording the size of thelesion, the organ location in which the lesion resides, and the imagenumber or slice position of the cross-sectional image depicting theidentified lesion.

Existing processes for tracking/analyzing lesions suffer from a numberof shortcomings. For example, manually outlining or tracing identifiedlesions to determine lesion size is a time-consuming process and issubject to variation due to different approaches taken by differentphysician reviewers. Furthermore, upon identifying an appropriate lesionfor analysis and determining the size thereof, a reviewing physicianoften manually records the organ location in which the lesion residesand the image number or slice position in which the lesion is depicted.

To track lesions over time, a reviewing physician typically navigatesthrough cross-sectional images captured at a later timepoint to find animage that depicts the same lesion analyzed previously. The reviewingphysician then repeats the processes of determining the size of thelesion and recording the organ location, often also recording imagenumber or slice position with or without the series number or name.Often, an additional step of preparing a report for oncological orpatient review must be performed by the reviewing physician or anotherentity.

Thus, conventional processes for tracking and/or reporting on lesionprogression over time can be laborious and/or time-intensive,particularly where numerous lesions were analyzed at the previoustimepoint. Because of the mundane nature of these processes, reviewingphysicians often become prone to dictation error when analyzing lesions,such as when recording lesion size, organ location, and/or image numberor slice position.

Furthermore, tumor response analysis may be performed independently bydifferent physicians focusing on different portions of a patient's body.Facilitating coordinated tumor response analysis by different physiciansand generating composite reports at that include analysis from differentspecialists is associated with several challenges.

There is a need for a method and/or system for determining and/orreporting an objective tumor response using cross-sectional medicalimages in a manner that improves reviewer speed, efficiency, and/oraccuracy.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY

Disclosed embodiments are directed to systems and methods for lesionanalysis. In some implementations, a system for facilitating lesionanalysis is configurable to identify a user profile associated with auser accessing the system. The user profile indicates a radiologyspecialty associated with the user. The system is also configurable toaccess a plurality of cross-sectional medical images associated with aparticular patient and identify a subset of cross-sectional medicalimages from the plurality of cross-sectional medical images thatcorrespond to the radiology specialty indicated by the user profile. Thesystem is also configurable to present the subset of cross-sectionalmedical images to the user in navigable form.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be apparent to one of ordinary skill inthe art from the description or may be learned by the practice of theteachings herein. Features and advantages of embodiments describedherein may be realized and obtained by means of the instruments andcombinations particularly pointed out in the appended claims. Featuresof the embodiments described herein will become more fully apparent fromthe following description and appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

To further clarify the above and other features of the embodimentsdescribed herein, a more particular description will be rendered byreference to the appended drawings. It is appreciated that thesedrawings depict only examples of the embodiments described herein andare therefore not to be considered limiting of its scope. Theembodiments will be described and explained with additional specificityand detail through the use of the accompanying drawings in which:

FIG. 1 illustrates a schematic representation of a system forfacilitating lesion analysis using one or more cross-sectional images,according to the present disclosure;

FIG. 2 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systempresents one or more cross-sectional medical images, according to thepresent disclosure;

FIG. 3 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemdisplays a different cross-sectional medical image in response to userinput, according to the present disclosure;

FIG. 4 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemreceives user input selecting a pixel region associated with a targetlesion represented in the displayed cross-sectional medical image,according to the present disclosure;

FIG. 5 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a zoomed view of the target lesion, according to the presentdisclosure;

FIG. 6 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a predicted shape of the target lesion, according to thepresent disclosure;

FIG. 7 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemdisplays major and minor axes based on the predicted shape of the targetlesion, according to the present disclosure;

FIG. 8 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemselectively disables a display of the predicted shape, according to thepresent disclosure;

FIG. 9 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a predicted shape of another target lesion, according to thepresent disclosure;

FIGS. 10-12 illustrate examples of display interface renderingsassociated with a system for facilitating lesion analysis as the systemreceives user input modifying the predicted shape the target lesion,according to the present disclosure;

FIG. 13 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemdisplays the predicted shape of the target lesion as modified by thereceived user input, according to the present disclosure;

FIG. 14 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides automatically determined location information for the targetlesion depicted in FIGS. 6-8, according to the present disclosure;

FIG. 15 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides automatically determined location information for anothertarget lesion, according to the present disclosure;

FIG. 16 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemreceives user input adding to the location information for the targetlesion previously provided by the system, according to the presentdisclosure;

FIG. 17 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides automatically determined location information for anothertarget lesion, according to the present disclosure;

FIG. 18 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemreceives user input modifying the location information for the targetlesion previously provided by the system, according to the presentdisclosure;

FIG. 19 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides updated location information for the target lesion based onreceived user input modifying previously provided location informationfor the target lesion, according to the present disclosure;

FIG. 20 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a lesion marker and location information for a non-targetlesion based on received user input, according to the presentdisclosure;

FIG. 21 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemdisplays a cross-sectional image including a representation of a lesionthat is not a target lesion or a non-target lesion; according to thepresent disclosure

FIG. 22 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a lesion marker and location information for the lesion that isnot a target lesion or a non-target lesion based on received user input,according to the present disclosure;

FIG. 23 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a list of analyzed lesions within a plurality ofcross-sectional medical images, according to the present disclosure;

FIGS. 24-26 illustrate examples of display interface renderingsassociated with a system for facilitating lesion analysis as the systemprovides a guided presentation of the analyzed lesions within theplurality of cross-sectional images, according to the presentdisclosure;

FIG. 27 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a report based on the analyzed lesions within the plurality ofcross-sectional images, according to the present disclosure;

FIG. 28 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides different sets of cross-sectional images captured at differenttimepoints, according to the present disclosure;

FIG. 29 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a predicted matching cross-sectional image that attempts tofind a matching target lesion that corresponds to a previously analyzedtarget lesion, according to the present disclosure;

FIG. 30 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemdisplays a modified predicted matching cross-sectional image afterreceiving user input modifying the predicted matching cross-sectionalimage, according to the present disclosure;

FIG. 31 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemdetermines shape and location information for the matching targetlesion, according to the present disclosure;

FIG. 32 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemdetermines a lesion marker for a matching non-target lesion thatcorresponds to a previously analyzed non-target lesion, according to thepresent disclosure;

FIG. 33 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemdetermines a lesion marker for a matching lesion that is not a targetlesion or a non-target lesion and that corresponds to a previouslyanalyzed lesion that was not a target lesion or a non-target lesion,according to the present disclosure;

FIG. 34 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a summary report based on one or more lesions analyzed atdifferent timepoints, according to the present disclosure;

FIG. 35 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a detailed table report based on one or more lesions analyzedat different timepoints, according to the present disclosure;

FIG. 36 illustrates an example of a display interface renderingassociated with a system for facilitating lesion analysis as the systemprovides a key image report based on one or more lesions analyzed atdifferent timepoints, according to the present disclosure;

FIG. 37 illustrates a conceptual representation of prompting a user toassociate cross-sectional images of a set of cross-sectional images withdifferent specialties;

FIG. 38 illustrates a conceptual representation of identifying andproviding different subsets of cross-sectional images for differentusers based on specialties associated with the users;

FIG. 39 illustrates a conceptual representation of generating orupdating a composite report based on marking, segmentation, or labelingperformed by different users for lesions represented in the differentsubsets of cross-sectional images; and

FIGS. 40-43 illustrate example flow diagrams depicting acts associatedwith facilitating lesion analysis.

DETAILED DESCRIPTION

While the detailed description may be separated into sections, thecontents within each section are not intended to be self-containeddescriptions and embodiments. Rather, the contents of each sectionwithin the detailed description are intended to be read and understoodas a collective whole where elements of one section may pertain toand/or inform other sections. Accordingly, embodiments specificallydisclosed within one section may also relate to and/or serve asadditional and/or alternative embodiments in another section having thesame and/or similar systems, modules, devices, methods, and/orterminology.

The embodiments disclosed herein will now be described by reference tosome more detailed embodiments, with occasional reference to anyapplicable accompanying drawings. These embodiments may, however, beembodied in different forms and should not be construed as limited tothe embodiments set forth herein. Rather, these embodiments are providedso that this disclosure will be thorough and complete, and will fullyconvey the scope of the embodiments to those skilled in the art.

Disclosed embodiments are directed to systems and methods for lesionanalysis. In one embodiment, a system for lesion analysis isconfigurable to perform various acts. In some embodiments, the actsinclude presenting a cross-sectional medical image to a user on adisplay, receiving user input directed to a pixel region correspondingto a lesion represented in the cross-sectional medical image,identifying a predicted shape of the lesion represented in thecross-sectional medical image based on contrast between the pixel regioncorresponding to the lesion and a surrounding pixel region,automatically determining location information for the lesion based onthe predicted shape of the lesion, and associating the locationinformation for the lesion with the lesion represented in thecross-sectional medical image or with the cross-sectional medical image.

In some embodiments, the acts include identifying a user profileassociated with a user accessing the system. The user profile indicatesa radiology specialty associated with the user. The acts also includeaccessing a plurality of cross-sectional medical images associated witha particular patient and identifying a subset of cross-sectional medicalimages from the plurality of cross-sectional medical images thatcorrespond to the radiology specialty indicated by the user profile. Theacts also include presenting the subset of cross-sectional medicalimages to the user in navigable form.

In some embodiments, the acts include identifying a user profileassociated with a user accessing the system. The user profile indicatessystem interaction preferences for the user. The system interactionpreferences for the user include an interaction presentation. The actsfurther include accessing a plurality of cross-sectional medical images,displaying the plurality of cross-sectional medical images to the userin navigable form within a user interface, and identifying a pluralityof controls within the user interface. The plurality of controlsincludes controls for: selecting a position within a pixel regioncorresponding to a lesion represented in the cross-sectional medicalimages, tracing the pixel region associated with the lesion representedin the cross-sectional medical images, and selecting locationinformation for the lesion. The acts further include associating atleast one the plurality of controls with the interaction presentationindicated in the system interaction preferences of the user profile,detecting user input operating the at least one of the plurality ofcontrols, and presenting the interaction presentation.

In some embodiments, the acts include accessing a first databasecomprising one or more entries including location information associatedwith one or more lesions represented in one or more cross-sectionalmedical images from a first plurality of cross-sectional medical imagesof a patient and displaying representations of each of the one or moreentries. The acts also include receiving user input selecting aparticular entry of the one or more entries. The particular entry isselected by either a user selection of the representation of theparticular entry or a user selection of a control for navigating to anext entry (where the particular entry is the next entry). The actsfurther include presenting the cross-sectional medical image and thelesion represented therein associated with the particular entry selectedby the received user input and rendering a lesion marker associated withthe particular entry overlaid on lesion represented in thecross-sectional medical image corresponding to the particular entry.

The methods and systems of the present disclosure are useful forevaluating tumor response to chemotherapy, targeted therapy,immunotherapy, radiation therapy, surgical therapy, ablative therapy,hyperthermia therapy, photodynamic therapy, laser therapy, gene therapy,biologic vector therapy, artificial vector therapy, and other forms oftherapy. Further, the methods and systems of the present disclosure areapplicable and useful to evaluate primary tumors, locoregional spread oftumors, and metastatic tumors; benign and malignant tumors; and avariety of tumor types, including: skin cancer, lung cancer, prostatecancer, breast cancer, colorectal cancer, kidney cancer, lymphoma,thyroid cancer, brain cancer, bone cancer, connective tissue cancer,muscle cancer, liver cancer, gastrointestinal cancer, pancreatic cancer,esophageal cancer, stomach cancer, melanoma, gynecologic cancer, cardiaccancer, and/or others.

Those skilled in the art will recognize that at least some of thepresently disclosed embodiments may solve at least some of the problemsassociated with conventional processes for analyzing and/or trackinglesions. For instance, by providing automated segmentation/shapeprediction of lesions and/or automated location identification forlesions, the disclosed systems and/or methods may allow for increasedspeed and/or efficiency when analyzing or tracking target lesions.Automated segmentation may also reduce variance caused by differentsegmentation approaches taken by different reviewing physicians whenanalyzing target lesions. Furthermore, automated location identificationcan similarly increase the speed of non-target lesion analysis and/ortracking.

Additionally, dictation errors may be avoided by automated recordationof key information (e.g., segmentation information, locationinformation, image number or slice location of key cross-sectionalimages including analyzed lesions) as disclosed herein, therebyimproving accuracy. Also, the disclosed embodiments may provide forincreased speed when tracking lesions over time by providingfunctionality for predicting matching cross-sectional images to identifya matching lesion at a later timepoint that corresponds to a lesion thatwas analyzed at a previous timepoint.

Furthermore, coordinated analysis from physicians with differentspecialties and/or subspecialties may be facilitated by selectivelyproviding subsets of cross-sectional images based on a specialty orsubspecialty associated with a particular user. Also, composite reportsmay be generated that include analysis from multiple physicians withdifferent specialties and/or subspecialties to provide comprehensivereports in an efficient and/or accurate manner. A composite report couldbe a patient-level report that includes a graph, table, key images, andstructured text that is a composite of information from all subspecialtyradiologists that evaluated images from the same patient. However, itshould be appreciated that in some embodiments, each subspecialty reportcan be generated at any time outside of its specific inclusion within acomposite report. That is, in some embodiments, the a patient-levelreport can include only one subspecialty report, and in someembodiments, a single subspecialty report can be generated by aphysician for review with the patient and/or for updating patient chartsand histories.

In addition, by providing personalized interaction presentationpreferences for practitioners reviewing cross-sectional images forlesion analysis as described herein, the mundanity of lesion analysismay be at least partially reduced, thereby potentially increasing theaccuracy of lesion tracking/analysis.

Thus, at least some embodiments of the present disclosure enable therapid identification and tracking of lesions (e.g., target lesions,non-target lesions, and/or other lesions) throughout a clinical trial ortreatment regimen so that reviewers can increase, speed, efficiency,and/or accuracy when analyzing lesions (e.g., over multiple timepoints)to evaluate tumor response.

Systems for Lesion Analysis

Referring now to FIG. 1, depicted is a schematic representation of asystem for lesion analysis (e.g., for determining an objective tumorresponse to an anti-cancer therapy) using one or more cross-sectionalimages, which can implement or serve as a basis for one or moreembodiments of the present disclosure. FIG. 1, generally, includes acomputing system 100 configured for use in lesion analysis. In a basicconfiguration, a computing system includes one or more hardwareprocessors and one or more hardware storage device that havecomputer-executable instructions stored thereon. The one or moreprocessors may execute the computer-executable instructions to cause thecomputing system to perform certain functions and/or operations. Acomputing system may be in wired or wireless communication (e.g., via anetwork) with one or more other devices, such as other computing systemsor processing centers, storage devices or databases, sensors or sensorsystems (e.g., cameras and/or imaging devices), etc. to facilitatecarrying out the operations detailed in the computer-executableinstructions. Additional details concerning components of computingsystems and computing environments will be described hereinafter.

Referring again to FIG. 1, a computing system 100 for carrying outlesion analysis is depicted as including various components, includinghardware processor(s) 108, hardware storage device(s) 112, I/O deviceinterface(s) 106, image processing module(s) 110, data processingmodule(s) 114, export module 118, primary database 116, and/or machinelearning module(s) 120. It will be appreciated, however, that a system100 for carrying out lesion analysis may comprise any number ofadditional or alternative components.

As used herein, the terms “executable module,” “executable component,”“component,” “module,” or “engine” can refer to any combination ofhardware components or software objects, routines, or methods that mayconfigure a computer system 100 to carry out certain acts. For instance,the different components, modules, engines, devices, and servicesdescribed herein may be implemented as objects or processors thatexecute on computer system 100 (e.g. as separate threads). While FIG. 1depicts several independent modules 110, 114, 118, 120, one willunderstand the characterization of a module is at least somewhatarbitrary. In at least one implementation, the various modules 110, 114,118, 120 of FIG. 1 may be combined, divided, or excluded inconfigurations other than that which is shown. For example, any of thefunctions described herein with reference to any particular module 110,114, 118, 120 may be performed by any number and/or combination ofprocessing units, software objects, modules, computing centers (e.g.,computing centers that are remote to computing system 100), etcetera. Asused herein, the individual modules 110, 114, 118, 120 are provided forthe sake of clarity and explanation and are not intended to be limiting.

The computing system 100 may obtain one or more cross-sectional medicalimages 102 for analysis of lesions represented in the cross-sectionalimages 102. The cross-sectional medical images may be captured by aradiologic device 104. In some implementations, the radiologic device104 and the computing system 100 are physically connected such that theone or more cross-sectional images 102 are transferred directly via thephysical connection. Alternatively, or additionally, the computingsystem 100 can obtain the cross-sectional images 102 indirectly via anetwork 128 to which both the radiologic device 104 and the computingsystem 100 are connected (whether via wired connections, wirelessconnections, or some combination), as known in the art. The network 128may be any number of private networks, such as an intranet of a hospitalor a private cloud or server, or the network 128 may be any number ofpublic networks, such as a public cloud or any other public networkaccessible via an internet connection.

The radiologic device 104 illustrated in FIG. 1 can include any medicalimaging device that generates cross-sectional images. By way ofnon-limiting example, a radiologic device 104 may comprise at least oneof: x-ray computed tomography (CT), computed tomography perfusion (CTP)imaging, positron emission tomography (PET), single-photon emissioncomputed tomography (SPECT), magnetic resonance imaging (MRI), orultrasound. Consequently, in some instances, the cross-sectional imagesmay include digital medical image data in the form of: CT images, CTPimages, PET images, SPECT images, MM images, or ultrasound images,respectively.

Upon obtaining the cross-sectional images 102, the computing system 100may store the cross-sectional images 102 in a primary database 116 or ahardware storage device 112 for immediate or later access and/or lesionanalysis. In some instances, at least some of the cross-sectional images102 are not stored on storage media that are local to the computingsystem 100 (e.g., primary database 116, hardware storage device(s) 112),but rather remain stored on remote computer-readable media such asstorage database system 124, hardware storage device(s) of a remotecomputing device 130 a, 130 b, 130 c, and/or any other remoterepository. Those skilled in the art will recognize that in such and/orother instances, the operations associated with lesions analysisdescribed herein referring to computing system 100 may be performed in adistributed and/or asynchronous manner by various computing devices.

As will be described in more detail with reference to FIGS. 2-39, thecomputing system 100 may operate singly or in combination with othercomputing systems (e.g., where at least some the cross-sectional images102 are stored remotely, and/or one or more modules described herein areassociated with a cloud service accessed by computing system 100) toanalyze one or more lesions represented in one or more of thecross-sectional images 102. The computing system 100 may render thecross-sectional images 102 utilizing one or more hardware processors 108(e.g., including a graphics processing unit (GPU)) for display on an I/Odevice interface 106, such as a monitor or other display screen. I/Odevice interface(s) 106 include any type of input or output device. Suchdevices include, but are not limited to, touch screens, displays, amouse, a keyboard, a controller, head-mounted displays, speakers,sensors, sensor systems, and so forth. Any type of input or outputdevice may be included among I/O device interface(s) 106, withoutlimitation.

Utilizing I/O device interface(s) 106, the computing system may receiveuser input directing the analysis of the cross-sectional images 102 andone or more lesions represented therein. For instance, a user mayoperate a mouse, keyboard, touchscreen, and/or other controller toselect a pixel or pixel region of a cross-sectional image 102 associatedwith a lesion represented in the cross-sectional image. In someinstances, the user may trace an outline, boundary, or shape of a lesionshown in a cross-sectional image. In other instances, the user mayselect, provide, and/or modify organ location information associatedwith a lesion under analysis. Additional examples and implementationdetails regarding user input received by the computing system 100 viaI/O device interface(s) 106 to facilitate lesion analysis (e.g., lesionidentification/marking, longitudinal analysis, report generation, etc.)will be described in more detail hereafter.

As described herein, the computing system 100 may automate variousaspects of lesion analysis to improve accuracy, reduce read times,and/or improve efficiency of lesion tracking and/or analysis. In someinstances, the computing system 100 utilizes image processing module(s)110 to automate segmentation of lesions identified in a cross-sectionalimage 102 to provide a predicted shape for the lesions. For example, inresponse to receiving user input (e.g., via I/O device interface(s) 106)selecting a pixel region within a lesion shown in a cross-sectionalimage 102, the image processing module 110 may analyze the intensity ofpixels within the pixel region. The image processing module 110 maydetermine that a boundary of the lesion exists where the contrastbetween pixels of the pixel region and pixels surrounding the pixelregion exceeds a predetermined threshold level. The image processingmodule 110 may provide a predicted shape of the lesion based on thedetected boundary, and the image processing module 110 may interpolatebetween boundary pixels to account for outlier boundary pixels and/or toprovide a smooth lesion boundary.

In some instances, the image processing module 110 utilizes multipledifferent contrast threshold levels or edge sensitivity levels todetermine multiple predicted shapes for the lesion, and the computingsystem 100 may allow or prompt the user to select a desired predictedshape as the segmentation for the lesion under analysis. In otherinstances, the threshold contrast or edge sensitivity is selectivelymodifiable by the user, and it will be appreciated that any otherconstraints may be applied to guide the segmentation process (e.g.,shape, size, contour, angular, and/or curvature constraints). By way ofexample, in some implementations, the image processing module 110 mayattempt to identify one or more (separate) pixel regions in neighboringcross-sectional images (e.g., at a higher or lower slice location orimage number) that correspond to the pixel region of the lesion selectedby the user, and perform contrast analysis on the separate pixel regionsof the neighboring images to determine predicted shapes for the separatepixel regions. The image processing module 110 may then utilize theshapes and/or sizes of the predicted shapes for the neighboring pixelregions as inputs for determining the predicted shape of lesion withinthe cross-sectional image under analysis.

As noted above, tracking target lesions over time is advantageous forobtaining accurate and precise evaluations of objective tumor response.To track a target lesion over multiple timepoints, a reviewer identifiesa previously analyzed target lesion within a set of cross-sectionalimages captured at a timepoint subsequent to the timepoint at which thetarget lesion was previously analyzed.

In some embodiments, the image processing module 110 at least partiallyautomates the identification of a later-timepoint cross-sectional imagethat includes the same lesion that was analyzed in a previous-timepointcross-sectional image. For example, the image processing module 110 mayidentify a predicted matching cross-sectional medical image (e.g.,within a later-timepoint set of cross-sectional images) that correspondsto a previously captured cross-sectional image that included a lesionthat was previously analyzed by image co-registration, feature matching,intensity similarity, and/or other techniques. The image processingmodule 110 may operate within various constraints to identify apredicted matching cross-sectional image, such as similarity thresholdsor a search window within which to search for a matching image (e.g., asearch window identified and/or centered based on a slice location ofthe previous-timepoint cross-sectional image). The image processingmodule 110 may expand the search window and/or selectively modify otherinputs and/or constraints if no later-timepoint cross-sectional imagemeets or exceeds a predefined threshold of similarity to theprevious-timepoint cross-sectional image containing the previouslyanalyzed lesion.

The computing system 100, as shown in FIG. 1, also includes machinelearning module(s) 120, which may be configured to perform any of theoperations, method acts, and/or functionalities disclosed herein. Forexample, machine learning module(s) 120 may comprise and/or utilizehardware components or computer-executable instructions operable tocarry out function blocks and/or processing layers configured in theform of, by way of non-limiting example, single-layer neural networks,feed forward neural networks, radial basis function networks, deepfeed-forward networks, recurrent neural networks, long-short term memory(LSTM) networks, gated recurrent units, autoencoder neural networks,variational autoencoders, denoising autoencoders, sparse autoencoders,Markov chains, Hopfield neural networks, Boltzmann machine networks,restricted Boltzmann machine networks, deep belief networks, deepconvolutional networks (or convolutional neural networks),deconvolutional neural networks, deep convolutional inverse graphicsnetworks, generative adversarial networks, liquid state machines,extreme learning machines, echo state networks, deep residual networks,Kohonen networks, support vector machines, neural Turing machines,and/or others.

As used herein, reference to any type of machine learning may includeany type of artificial intelligence algorithm, device, structure, and/orarchitecture. Any amount or type of training data (e.g., datasetscomprising cross-sectional medical images, control inputs provided byusers, and/or, as ground truth, data corresponding to lesion analysis(e.g., lesion identification, segmentation, etc.) performed using thecross-sectional medical images) may be used (and/or later refined) totrain a machine learning model to provide output for facilitating any ofthe disclosed operations.

In some instances, the computing system 100 utilizes machine learningmodule 120 to at least partially automate the localization of targetlesions and/or non-target lesions. In some implementations, the machinelearning module 120 is trained to identify location information for alesion based on various input (e.g., type of cross-sectional image underanalysis). For example, in some implementations, the computing system100 provides the predicted shape (e.g., as determined above utilizingthe image processing module 110, and/or as modified/indicated by userinput) to the machine learning module 120 as input and causes themachine learning module to identify the location information for theanalyzed lesion based on the predicted shape.

It should be noted that the machine learning module 120 may also betrained to receive other input for identifying location information fora lesion. In some instances, the machine learning module 120 receives asinput a form of metadata indicative of an anatomical or organ locationof the lesion. Such metadata may be associated with the particularcross-sectional image under review, the set of cross-sectional images102 of which the particular cross-sectional image is a part, or even auser profile associated with the user performing the lesion analysis.For example, cross-sectional image or image set metadata may include anidentifier of a slice location or image number or applicable anatomicallocation for the images captured (e.g., chest, abdomen, head, neck).Also, the user profile of the reviewer may indicate a radiologysubspecialty (e.g., neuroradiology or thoracic radiology, which caninclude chest or abdomen subspecialties) which may inform theidentification of the anatomical information associated with the lesionunder analysis. In other instances, the machine learning module 120receives as input pixel coordinates of user input directed at the lesionor of a pixel region within the lesion to guide the identification ofthe location information for the lesion. In yet other instances, themachine learning module analyzes structures neighboring the identifiedlesion and/or analyzes the cross-sectional image as a whole to identifythe location information for the identified lesion.

As depicted in FIG. 1, the computing system 100 also includes dataprocessing module(s) 114 and an export module 118. The data processingmodule 114, in some implementations, operates to determine or obtainlesion metrics associated with a lesion (e.g., a target lesion) underanalysis or review. For instance, the data processing module 114 may,for one or more lesions at one or more timepoints, determine a majoraxis, a minor axis, and/or pixel area based on predicted lesion shape.The data processing module 114 may also perform calculations on lesionaxes (e.g., comparing sums of the lengths of lesion axes over time) orother metrics to determine tumor response and/or disease progressionbased on predefined tumor response criteria as discussed above.

The data processing module 114 and/or the export module 118, in someimplementations, is also responsible for organizing and/or storingdata/information associated with analyzed lesions. For example, the dataprocessing module 114 may store and/or copy within one or more lists ordatabases the predicted shape, axes (major and/or minor), slice locationor cross-sectional image number, location information, key images (e.g.,images showing close-up views of a lesion), or any combinations orrepresentations thereof associated with any number of lesions at anynumber of timepoints. For example, in some embodiments, any of theforegoing types of data associated with the lesions become stored inassociation with and/or within the cross-sectional images themselves(e.g., as metadata or as a modified version of the cross-sectionalimages with data implemented or embedded therein). In some instances,the data/information become stored within hardware storage device(s)112, remote storage database system(s) 124 (e.g., within a cloudserver), and/or on one or more remote computing device(s) 130 a, 130 b,130 c (via network 128).

In some implementations, and as will be discussed hereafter, the dataprocessing module 114 and/or export module 118 may compile or generatereports based on any of the data/information described herein foroncological and/or patient review. Such reports may comprise one or moreresults and/or output of lesion analysis performed by one or more thanone physician.

It will be appreciated that the computing devices 130 a, 130 b, 130 ccan have any or all of the components and modules described above forthe general computing system 100. In some instances, the computingsystem 100 can include the workstation of a physician reviewer.Alternatively, the computing system 100 can include a server for hostingor facilitating user interaction with cross-sectional images and/orcomputer-executable instructions (e.g., in the form of software or anSaaS platform) for standardizing target lesion identification andselection within cross-sectional images, as described herein. Similarly,the computing devices 130 a, 130 b, 130 c can represent the workstationsof other reviewers, or the computing devices 130 a, 130 b, 130 c can beuser profiles or virtual instances of computing system 100. Forinstance, different physician reviewers with different specialtiesand/or subspecialties may perform lesion analysis on different subsetsof one or more sets of cross-sectional medical images, and such analysismay be performed at the same or different times. Such analysis bymultiple reviewing physicians may be compiled into a composite report byany of the computing systems/devices described herein.

Regardless of the physical and/or virtual organization of the computingsystem 100 and/or the associated computing devices 130 a, 130 b, 130 c,embodiments of the present disclosure enable cross-sectional images tobe received and/or viewed at any of the foregoing system/devices 100,130 a, 130 b, 130 c. The ellipsis shown in FIG. 1 indicate that anynumber of computing systems (e.g., 1, 2, or more than 3) may be incommunication with computing system 100 via network 128.

Example Implementations of Lesion Analysis

The following discussion refers to FIGS. 2-39 and provides additionaldetails, examples, and implementations related to systems and methodsfor analyzing lesions in cross-sectional medical images. It will beappreciated that the contents of the accompanying Figures are notmutually exclusive. For instance, any feature, component, or embodimentshown in any one of the accompanying Figures may be combined with one ormore features, components, or embodiments shown in any otheraccompanying Figure.

FIG. 2 illustrates an example of a display interface renderingassociated with a system for lesion analysis (e.g., computing system100) as the system presents cross-sectional medical images within a userinterface. The display interface rendering may be displayed to a user onone or more I/O device interfaces 106 (e.g., on a screen or monitor). Asshown, the display interface rendering includes a rendering of across-sectional medical image 202, which may comprise onecross-sectional image of a set or plurality of cross-sectional medicalimages (i.e., the rendered image is slice 1 of 273, as indicated in theupper-left corner of the rendered image) that is available to the user.The cross-sectional medical images may be captured by a radiologicdevice 104, as described above, and may be associated with a particularpatient.

As indicated hereinabove with reference to FIG. 1, the user may accessthe system through a local connection with the system (e.g., byinterfacing with the system itself or through a local network to whichthe system is connected) or a remote connection with the system (e.g.,via a cloud service).

The user interface includes an icon 204 representative of auser-operated controller (shown in FIG. 2 as a mouse cursor) and/orvarious controls (e.g., buttons of a mouse that controls the mousecursor) for performing various functions. Other controls may be operablefor providing user input within the user interface not shown in FIG. 2(e.g., keyboard or other physical controls that are operable by the userto interface with the system).

FIG. 3 illustrates an example of a display interface rendering as thesystem displays a different cross-sectional medical image 302 inresponse to user input. In particular, in FIG. 3, the system displaysslice 20 of 273 (whereas slice 1 of 273 was presented as shown in FIG.2). The user may provide input at an I/O device interface 106 (e.g., byscrolling a mouse wheel or moving the mouse cursor while operating acontrol) to navigate from one cross-sectional medical image to anotherin order to view different cross-sectional medical images. By navigatingthrough the cross-sectional images provided in the set, a user (e.g., areviewer) may navigate to a suitable lesion (e.g., a target lesion or anon-target lesion) represented in a cross-sectional image upon which tocarry out lesion analysis, as described herein.

FIG. 4 illustrates an example of a display interface rendering as thesystem presents the cross-sectional medical image 302 (e.g., from FIG.3) and receives user input selecting a pixel region 402 associated witha target lesion 404 represented in the displayed cross-sectional medicalimage 302. In the depicted example, the user has triggered a control forselecting a target lesion 404 for which to measure lesion metrics (e.g.,axis length, pixel area, location information, etc.), causing, in thisexample, the mouse cursor 406 to change shape (e.g., to a cross ratherthan an arrow as shown in FIGS. 2 and 3). In the example shown in FIG.4, the cursor 406 has navigated over a lesion 404 in the axillary regionto provide user input directed to a pixel region 402 within the lesion404 represented in the cross-sectional image 302. The user may providethe user input via an I/O device interface, such as by pressing a mousebutton.

FIG. 5 illustrates an example of a display interface rendering as thesystem provides a zoomed view of the target lesion 404 within thecross-sectional medical image 302. In some embodiments, in response tothe user input described with reference to FIG. 4 (e.g., pressing amouse button), the system zooms (e.g., automatically) the presentationof the cross-sectional medical image 302 toward the lesion 404represented in the cross-sectional medical image 302 that the user inputwas directed toward (e.g., toward the pixel or pixel region 402 selectedby the user input). The zoomed presentation may provide for moreaccurate and/or rapid analysis of the lesion 404 by enabling users tomore easily recognize the features of the analyzed lesion 404. In someinstances, the system auto-zooms so that the selected target lesion 404is centered (or as nearly centered as possible given the location of thelesion) in the viewing area.

In some implementations, the degree of magnification is modifiable. Forinstance, the degree of magnification may be automatically modifiedbased on the size and/or location of the selected target lesion 404. Insome instances, the degree of magnification can be adjusted by a user(e.g., according to their individual user profile or other settings). Byway of non-limiting example, the image may be magnified within a rangeof about 100% to about 500% or more. In some embodiments, a standard orpreset magnification setting is 200%.

Following segmentation and/or further identification of the targetlesion 404 (as will be described in more detail hereinbelow), the systemcan return the magnification and view to the original 100% view of thecross-sectional medical image 302 (e.g., as shown in FIG. 4). Theforegoing can be done automatically or in response to user input. Insome embodiments, the magnification/zoom feature is performedautomatically by the system to maximize the ease and speed oftraditional radiology workflows.

FIG. 6 illustrates an example of a display interface rendering as thesystem provides a predicted shape 602 of the target lesion 404. Thepredicted shape 602 may be thought of as a predicted segmentation forthe lesion 404 selected by the user. In some embodiments, the systemprovides the predicted shape 602 utilizing, at least in part, imageprocessing module(s) 110 and/or machine learning module(s) 120 based oncontrast or intensity analysis of one or more cross-sectional imagesand/or other factors/inputs as described hereinabove with reference toFIG. 1. In the example depicted in FIG. 6, the system presents arendering of the predicted shape 602 of the lesion 404 overlaid on thelesion 404 as represented in the cross-sectional image 302 to the user.By providing automated segmentation (e.g., shape prediction), at leastsome presently disclosed embodiments may increase the rate at whichlesions are analyzed in cross-sectional medical images.

FIG. 7 illustrates an example of a display interface rendering as thesystem displays a major axis 702 and a minor axis 704 based on thepredicted shape 602 of the target lesion 404. For instance, the systemmay calculate major axis 702 and/or the minor axis 704 for the analyzedlesion 404 utilizing, at least in part, data processing module(s) 114and/or machine learning module(s) 120. Upon determining the major axis702 and/or the minor axis 704, the system may display one or both ofthem overlaid on the lesion 404 in the cross-sectional image 302, asillustrated in FIG. 7. The respective lengths of the major axis 702 andthe minor axis 704 may also be presented by the system, as shown in FIG.7.

FIG. 8 shows an example of a display interface rendering as the systemselectively disables a display of the predicted shape 602 of the targetlesion 404, while maintaining a display of the major axis 702 and theminor axis 704 associated with the lesion 404. The system mayselectively disable the presentation of the predicted shape 602 inresponse to received user input (e.g., to allow the user to rapidlyverify that the segmentation provided by the system for the targetlesion 404 is desirable or correct). Those skilled in the art willappreciate, in view of the present disclosure, that the system need notcalculate and/or present major and/or minor axes for all analyzedlesions. For instance, for some types of lesions, the minor axis may beconsidered more critical for analyzing tumor response (e.g., whenanalyzing lymph nodes) so calculations and/or presentations of the majoraxis may be omitted.

The system may prompt the user to accept or reject the predicted shape602 or segmentation provided by the system, as indicated in FIG. 8 bythe right-facing arrow button for accepting the predicted shape 602 andthe left-facing arrow button for rejecting the predicted shape 602 shownin/over the upper-left portion of the displayed cross-sectional image302.

In some instances, the system (e.g., utilizing image processingmodule(s) 110 and/or machine learning module(s) 120) may provide anundesirable predicted shape or segmentation for a lesion selected byuser input (e.g., due to variations in quality/resolution ofcross-sectional medical images). For instance, FIG. 9 illustrates anexample of a display interface rendering as the system displays across-sectional medical image 902 (e.g., image slice 233 of 273) andprovides a predicted shape 904 of a target lesion 906 proximate to thepelvis of the patient (e.g., in response to user input). As shown, thesystem presents a rendering of the predicted shape 904 or segmentationand the major axis 908 and the minor axis 910 overlaid on the targetlesion 906 within the cross-sectional image 902 and prompts the user toaccept or reject the predicted shape 904 or segmentation (indicated bythe buttons shown over the upper-left portion of the cross-sectionalimage 902).

Although the major axis 908 and the minor axis 910 generated by thesystem based on the predicted shape 904 may appear correct, the user maybelieve that a more desirable segmentation of the lesion 906 may beachieved manually. Thus, in some instances, the user may reject thepredicted shape 904 generated by the system and/or provide user inputthat modifies the predicted shape 904 or segmentation for the targetlesion 906.

FIGS. 10-12 illustrate examples of display interface renderings as thesystem receives user input modifying/defining the predicted shape thetarget lesion 906. In some embodiments, as depicted, the system providesa free-form region of interest trace tool to allow the user to modifythe segmentation of the lesion 906. For example, FIGS. 10-12 show that auser may control an icon 1002 (e.g., by selecting a mouse button andmoving the mouse, or by operating a touch screen, etc.) to manuallytrace a line 1004 that modifies or defines a predictedshape/segmentation for the target lesion 906.

Based on the user input modifying/defining the predictedshape/segmentation for the target lesion 906, the system may provide anupdated predicted shape 1302 of the target lesion 906 (and/or a majoraxis 1304, a minor axis 1306, and/or other lesion metrics for the targetlesion 906), as shown in FIG. 13. In some instances, the system utilizesat least a part of the user input modifying the predicted shape (e.g.,as shown in FIGS. 10-12) as an input to generate the updated predictedshape 1302 of the target lesion 906 (e.g., as an additionalconstraint/input that affects the boundaries of the updated predictedshape 1302), while in other instances the system receives user inputcorresponding to a complete trace of the target lesion 906 such that thesystem makes no further predictions about the shape and accepts the userinput as the updated predicted shape 1302 or updated segmentation of thetarget lesion 906.

FIG. 14 illustrates an example of a display interface rendering as thesystem provides automatically determined location information 1402 forthe target lesion 404 in the axillary region shown and described withreference to FIGS. 6-8. A system may utilize any combination ofcomponents (e.g., image processing module(s) 110, data processingmodule(s) 114, machine learning module(s) 120, hardware processor(s)108) to generate location information 1402 for a target lesion.

As shown, the location information 1402 includes an indication region1404 of whether the lesion under analysis is a mass or a lymph node, anindication region 1406 of whether the lesion is located within the neck,chest, abdomen, pelvis, arm, or leg of the patient, as well as otherindication regions 1408 of the anatomical location of the lesion. In theembodiment shown, the system automatically determined that the lesion404 is a lymph node located in the left axillary region of the chest ofthe patient. This information is reflected in the location information1402 presented by the system contemporaneously with a presentation of arendering of the lesion 404 in the cross-sectional image 302 (as well asthe predicted shape/segmentation of the lesion 404 and the major andminor axes of the lesion). For instance, indication region 1404 of thelocation information 1402 has the label/button “Lymph Node” emphasized,selected, or highlighted; indication region 1406 of the locationinformation 1402 has the label/button “Chest” emphasized, selected, orhighlighted; and indication regions 1408 of the location information1402 have the labels/buttons of “Axillary” and “L” (indicating “Left”)emphasized, selected, or highlighted.

In some instances, as shown in FIG. 14, the system presents the locationinformation 1402 to the user and allows the user to accept or modify thelocation information 1402 provided by the system. For instance, FIG. 14shows a check mark button shown over the upper-left portion of therendered cross-sectional image 302, which a user may select upondetermining that the location information 1402 generated by the systemappears accurate or desirable. Furthermore, the various elements of theindication regions 1404, 1406, 1408 of the location information 1402may, in some implementations, be presented as selectable buttons,enabling a user to easily modify the location information 1402 for atarget lesion.

As noted above with reference to FIG. 1, the system may utilize (atleast in part) machine learning module(s) 120 to automatically determinelocation information for the lesion. The machine learning module 120 maybe trained to identify location information based on an input of apredicted shape/segmentation of the lesion and/or coordinates of userinput selecting the lesion (described above), or the machine learningmodule 120 may be trained/configured to utilize other additional oralternative inputs such as other structures present in thecross-sectional image or other characteristics of the cross-sectionalimage as a whole, metadata associated with the cross-sectional images, auser profile, and/or other identifiers.

FIG. 15 illustrates an example of a display interface renderingassociated with a system for lesion analysis as the system presents across-sectional medical image 1502 (slice 46 of 273) and providesautomatically determined location information 1506 for a target lesion1504 in the mediastinal region. As noted above, the system may presentthe automatically determined location information 1506 for the targetlesion 1504 and prompt the user to either accept the locationinformation 1506 or modify the location information 1506. In the exampleshown, the location information 1506 includes an indication region 1508that indicates that the lesion 1504 being analyzed is a lymph node, anindication region 1510 that indicates that the lesion 1504 is located inthe chest of the patient, and an indication region 1512 that indicatesthat the lesion 1504 is located in the mediastinal region of the chest.

However, the location information 1506 also includes an indicationregion 1514 with no particular element thereof emphasized, selected, orhighlighted, thereby prompting the user to make a selection among theelements of the indication region 1514 to modify the locationinformation 1506. Here, the system prompts the user to specify withinthe indication region 1514 of the location information 1506 whether thetarget lesion 1504 is a superior, prevascular, upper or lowerparatracheal, AP window, subcarinal, paraesophageal, paraaortic,internal mammary, or cardiophrenic lymph node within the mediastinalregion. Accordingly, in some implementations, the system provideslocation information to a certain level of granularity but allows theuser to add additional details if the user desires (e.g., where thesystem is unable to provide a granular location estimate).

FIG. 16 illustrates an example of a display interface rendering as thesystem receives user input within the indication region 1514 of thelocation information 1506, where the user input indicates that thetarget lesion 1504 identified with reference to FIG. 15 is a lowerparatracheal lymph node in the mediastinal region of the chest. Inresponse to this user input, as shown in FIG. 16, the system generatesan additional indication region 1602 and further prompts the user toinput whether the target lesion 1504 is “left” or “right,” and the usermay elect whether to so specify before choosing to accept or reject thelocation information provided by the system and/or as modified by theuser (e.g., using the check mark button in the upper-left region toaccept, or the leftward arrow button in the upper-left region toreject). As will be described hereinafter, location information for atarget lesion, whether solely generated by the system or at leastpartially modified by a user, may be stored for use in lesion analysis.

In some instances, the system provides location information that is atleast partially incorrect. For instance, FIG. 17 illustrates an exampleof a display interface rendering as the system displays across-sectional image 1702 (slice 156 of 273) and provides automaticallydetermined location information 1706 for a target lesion 1704 that is alymph node located in the mesenteric region of the patient. In theinstance shown, the system provides location information 1706 thatincludes an indication region 1708 indicating that the target lesion1704 is a mass, an indication region 1710 indicating that the targetlesion 1704 is located within the abdomen of the patient, an indicationregion 1712 indicating that the target lesion 1704 is located in theliver, and an indication region 1714 allowing the user to selectadditional location information for the target lesion 1704. As notedabove, the various elements of the various indication regions 1708,1710, 1712, and 1714 of the location information 1706 may be provided asselectable buttons, enabling a user to modify the location information1706 before accepting or rejecting the location information 1706 (e.g.,via the check mark button or the leftward arrow button in the upper-leftregion).

The user may determine that the location information 1706 provided bythe system is at least partially inaccurate in the particular instanceand choose to modify the location information 1706. FIG. 18 illustratesan example of a display interface rendering as the system receives userinput modifying the location information 1706 for the mesenteric lymphnodes target lesion 1704 previously provided by the system. As shown,the cursor 1802 has been navigated (e.g., via a mouse or other interfacedevice) over the selectable “Lymph Node” element/button of theindication region 1708 of the location information 1706 to provide userinput to the system to modify the location information 1706 associatedwith the target lesion 1704 to indicate that the target lesion is alymph node rather than a mass.

FIG. 19 illustrates an example of a display interface rendering as thesystem responds to the received user input described with reference toFIG. 18 (e.g., selecting the “Lymph Node” element/button of theindication region 1708 of the location information 1706). As shown inFIG. 19, the system provides updated location information 1902 for thetarget lesion 1704. For example, in some implementations, the systemautomatically determines the updated location information 1902 byproviding the predicted shape of the target lesion 1704 and/or at leasta portion of the initial location information 1706 as modified by theuser (e.g., specifying “Lymph Node”) as input to a machine learningmodule 120 to cause the machine learning module 120 to identify updatedthe location information 1902 for the lesion 1704. Put differently, atleast a portion of the user modifications to the initial locationinformation 1706 may act as an additional constraint or input for theidentification process performed by the machine learning module 120 todetermine the updated location information 1902.

In the example shown in FIG. 19, based at least in part on the receiveduser input (e.g., specifying “Lymph Node”), the system generates theupdated location information 1902 for the target lesion 1704, includingan indication region 1904 that the target lesion 1704 is a lymph node,an indication region 1906 that the target lesion 1704 is in the abdomenof the patient, an indication 1908 that the target lesion 1704 is in themesenteric region of the abdomen, and an indication region 1910 allowingthe user to further select from among “small bowel,” “colon,” or“sigmoid.” After determining that the updated location information 1902is desirable/acceptable (whether additional modifications are performedor not), the user may choose to accept the updated location information1902 (e.g., by selecting the check mark button in the upper-leftregion).

Upon establishing location information for a lesion, whether in fullyautomated or partially automated fashion, the system may associate thelocation information with the lesion represented in the cross-sectionalmedical image and/or with the cross-sectional medical image itself(e.g., at least partially utilizing data processing module(s) 114),thereby allowing the location information to be used for lesionanalysis.

Although FIGS. 2-19 have focused, in at least some respects, on analysisof target lesions, it will be appreciated that at least some of theprinciples disclosed herein are applicable to non-target lesions orother findings as well. For instance, FIG. 20 illustrates an example ofa display interface rendering associated with a system for lesionanalysis as the system displays a cross-sectional image 2002 (slice 22of 273) and illustrates a lesion marker 2004 and location information2006 for a non-target lesion 2008 based at least in part on receiveduser input. For instance, in the example shown, the system received userinput electing to identify a non-target lesion and user input directedto another axillary lymph node (e.g., user input selecting a pixelregion associated with the non-target lesion 2008). The system, inresponse, may utilize coordinate information associated with user inputselecting the non-target lesion (or the pixel region within thenon-target lesion 2008) and/or other information (e.g., image slicenumber) to generate location information 2006 for the non-target lesion2008. In some instances, the system identifies a predicted shapeassociated with the selected non-target lesion 2008 to generate thelocation information 2006 at least in part based on the predicted shape.In the depicted example, the system presents the location information2006 indicating that the non-target lesion 2008 is a lymph node locatedin the axillary region of the chest and prompts the user to accept ormodify the location information.

As shown, the system refrains from rendering the predicted shape or anyminor and/or major axes that could be derived therefrom overlaid on thecross-sectional medical image 2002 (e.g., to facilitate faster analysisof non-target lesions). Rather than minor and/or major axes and/orpredicted shape/segmentation information, the system simply provides thelesion marker 2004 in the form of an “X” over the non-target lesion 2008to indicate the location of the non-target lesion 2008 within thecross-sectional image 2002 (e.g., for review at subsequent timepoints,as discussed hereinafter). In addition, in at least some instances, thesystem provides less granular location information for non-targetlesions than it does for target lesions, and the system may providelocation information that generalizes at least some location details fornon-target lesions (e.g., to facilitate faster analysis of non-targetlesions). Accordingly, those skilled in the art will recognize thatmachine learning module(s) 120 and/or other components of the presentdisclosure may include different algorithms for identifying locationinformation for target lesions and non-target lesions.

In some implementations, the system is configured to track lesions orother findings aside from target and non-target lesions. For instance,FIG. 21 illustrates an example of a display interface rendering as thesystem displays a cross-sectional image 2102 (slice 115 of 273)including a representation of a lesion 2104 that is not a target lesionor a non-target lesion. The mouse cursor 2106 is positioned over thelesion 2104 of interest. FIG. 22 illustrates an example of a displayinterface rendering as the system displays a lesion marker 2202 andlocation information 2204 for the lesion 2104 of interest shown in FIG.21. The lesion marker 2202 may positionally correspond with the userinput received selecting/identifying the lesion of interest (e.g.,selection of a pixel region within the cross-sectional image 2102), andthe location information 2204 may be entirely provided by user input(e.g., utilizing selectable buttons or providing text input, as shown inFIG. 22). Providing functionality for identifying a lesion of interestthat is not a target lesion or a non-target lesion may allow forreviewers to force subsequent reviewers to review such lesions ofinterest, which may provide a more comprehensive lesion analysis overmultiple timepoints.

In some embodiments, the system is configured to prompt the user tocharacterize the other finding(s) according to established clinicalguidelines for indeterminate or incidental findings from literature, aclinical or radiologic society, or published national or internationalguidelines. Examples may include characterizing other findings accordingto the American College of Radiology (ACR) Reporting and Data Systems(ACR RADS) including BI-RADS, C-RADS, HI-RADS, LI-RADS, Lung-RADS,NI-RADS, O-RADS, PI-RADS, or TI-RADS. For example, the system may promptthe user to characterize a lung nodule per Lung-RADS or by other methodssuch as Fleischner Criteria. The system may prompt the user tocharacterize any incidental finding using established criteria ormethods known in the art. For example, the system could prompt the userto characterize an adrenal nodule following guidelines from the ACRIncidental Finding Committee, including white papers from a society orestablished group of experts. Furthermore, the system could assist theuser by providing recommendations on follow up or management of theincidental findings using these same or newly established guidelines.

As noted above, information related to lesion location, predicted lesionshape/segmentation, lesion axes, lesion pixel area, slice location orimage number of the cross-sectional image depicting the lesion, and/orother lesion metrics may be stored (e.g., utilizing data processingmodule(s) 114) in computer-readable storage (e.g., within hardwarestorage device(s) 112, primary database 116, etc.). Such information maybe stored in any suitable form, such as a series of entries in a list orother data structure.

In some implementations, the system is configurable to displayrepresentations of the stored lesion information contemporaneously withcross-sectional images of the set of cross-sectional images that includethe analyzed lesions.

FIG. 23 illustrates an example of a display interface rendering as thesystem provides a rendering of a list 2302 of information related toanalyzed lesions within the plurality of cross-sectional medical imagesalong a left side of the user interface. As shown, the list 2302includes at least some information for analyzed lesions, such asabbreviations indicating location information for each lesion and/or thelength of axes (if any) associated with each lesion. The list 2302 maypersist within the user interface during lesion analysis and may bedynamically updated in real time to reflect newlyidentified/classified/labeled lesions.

In some instances, the system provides functionality for a guidedpresentation of the tracked lesions, and this guided presentation may bedriven at least in part by the representations of the stored lesioninformation (e.g., for error checking purposes or to check the lesionanalysis of another reviewer). Providing a guided presentation ofanalyzed lesions may greatly expedite processes for reviewing the user'sown lesion analysis or the lesion analysis of other reviewers,particularly where several lesions were analyzed in one or more sets ofcross-sectional images. Providing a guided presentation of analyzedlesions may also expedite processes for performing lesion analysis oncross-sectional images taken at different timepoints, as describedhereinafter. In the example list 2302 shown in FIG. 23, the user mayinitiate the guided presentation of the analyzed lesions by providinguser input selecting the “Start Autopilot” button 2304 in the userinterface.

FIGS. 24-26 illustrate examples of display interface renderings as thesystem provides a guided presentation of the analyzed lesions within theplurality of cross-sectional images. In the example shown in FIG. 24,the user has provided the user input indicated in the description ofFIG. 23 (i.e., selecting the “Start Autopilot” button 2304), and thesystem, in response, has navigated to the cross-sectional image 302corresponding to the first entry 2402 in the list 2302 of analyzedlesions shown in the left side of the user interface. As shown, thefirst entry 2402 corresponds to the segmented axillary target lesion 404shown and described with reference to FIGS. 6-8 and 14 hereinabove, sothe system navigates to the cross-sectional image 302 that depicts thesegmented axillary target lesion 404 and renders the applicablecross-sectional image 302 with the lesion predicted shape and major andminor axes overlaid on the target lesion 404 represented in thecross-sectional image 302 (major and/or minor axes associated with alesion, whether a target lesion, a non-target lesion, or other finding,may be regarded as “lesion marker(s)”). Accordingly, a reviewer mayquickly navigate to the first analyzed lesion within the set ofcross-sectional images.

FIG. 24 shows that the user interface displayed may includefunctionality for readily allowing a reviewer to modify lesioninformation that has been stored in or in association with the list2302. For instance, the user interface shown in FIG. 24 includes a“Redraw” button 2404, which may allow a reviewer to discard or modifylesion predicted shape and/or lesion location information associatedwith the list 2302. Such functionality may allow a reviewing physicianto review lesion analysis performed previously (e.g., whether by thesame reviewer or another reviewer) in a rapid and/or efficient manner. Auser interface may also include functionality for discarding one or moreentries of the list 2302.

FIG. 24 also shows that in some instances, during a guided presentationof analyzed lesions, the user interface provides controls for navigatingto a next cross-sectional image in the list 2302 that includes ananalyzed (or marked) lesion, as indicated by the “Next” button 2406.Accordingly, a user may provide input for navigating to and rendering anext cross-sectional image within the set of cross-sectional images forthe particular patient that includes an analyzed or tracked lesion(e.g., as recorded in the list 2302). A user interface may also includea button or other interface element for navigating to a previouscross-sectional image that includes an analyzed (or marked) lesion(e.g., as recorded in the list 2302).

FIG. 25 shows a display interface rendering provided by the system afterreceiving user input activating a control for navigating to the nextcross-sectional medical image in the list 2302 that includes a markedlesion (e.g., by selecting the “Next” button 2406). In response, thesystem navigates to and displays the next cross-sectional medical imagethat includes a marked lesion corresponding to the second entry 2502 inthe list 2302, along with any applicable segmentation/axis dataassociated with the analyzed lesion. In the instance shown in FIG. 25,the second entry 2502 in the list 2302 is associated with the segmentedmediastinal target lesion 1504 shown and described with reference toFIGS. 15-16.

In some instances, the system may provide functionality for allowing theuser to navigate directly to any cross-sectional image associated withthe list 2302 containing an analyzed lesion. For instance, a system maybe configured to receive user input that selects one of the entries ofthe list 2302 (e.g., which entries may comprise representations ofstored lesion metrics/data/images). Such user input may comprisenavigating a mouse cursor over and selecting one of the entries of thelist 2302, as shown in FIG. 26, which shows an icon 2602 positioned overand selecting an entry 2604 of the list 2302. The entry 2604 of the list2302 selected in FIG. 26 comprises a representation of the storeddata/information (e.g., location information, cross-sectional image,location within the cross-sectional image, etc.) associated with theaxillary non-target lesion 2008 shown and described with reference toFIG. 20.

In response to the user input selecting entry 2604 of the list 2302, thesystem renders and presents the non-target lesion 2008 with the lesionmarker 2004 (e.g., an “X” marker) indicating the location of thenon-target lesion 2008 within the cross-sectional image 2002. It will beappreciated that the direct navigation described herein with referenceto FIG. 26 may also be applied to selections of entries of the list 2302that are associated with target lesions or other lesions/findings, aswell as non-target lesions, and the system may, in some instances,display additional or different information associated with such lesions(e.g., segmentation information, major and/or minor axes, etc.).

In some embodiments, in addition to providing functionality for a guidedpresentation of analyzed lesions, the system may also utilize the storedlesion metrics, data, and/or information (e.g., associated with entriesof the list 2302) to generate reports. Such reports may be used, forinstance, for oncological and/or patient review. FIG. 27 illustrates anexample of a display interface rendering as the system provides report2702 based on the analyzed lesions within the plurality ofcross-sectional images. The report includes various information, such asa listing of target lesions 2704 (including composite length metrics forthe target lesions), a listing of non-target lesions 2706, and a listingof other findings 2708, as well as information for each class ofanalyzed lesion (e.g., axis length for target lesions, status fornon-target lesions, notes related to other findings, etc.).

The report 2702 of FIG. 27 also includes a graphical representation 2710illustrating the composite length of the analyzed target lesions. Thecomposite length may comprise a summation of various lengths of variousanalyzed target lesions. As noted above, a target lesion may have aminor axis and a major axis associated therewith, and the system mayselect from among the minor and major axes for contribution to thecomposite length depending on the tumor response criteria (e.g., undersome tumor response criteria, the system may select minor axes for lymphnodes, and major axes for masses). The report 2702 also includes anindication of the tumor response criteria that informs the data/analysisprovided in the report 2702, and reports may vary in form and/or contentbased on the tumor response criteria selected or applicable.

The report 2702 shown in FIG. 27 is related to lesion analysis performedfor lesions present in cross-sectional images of a patient associatedwith a single timepoint (e.g., a single imaging session). The report2702 also includes information showing the reader who performed thelesion analysis, as well as a date for the analysis or the imaging.Additional details concerning lesion analysis reports, including reportsinvolving lesions analyzed at multiple timepoints and/or by multiplereaders, will be provided hereinafter. Additional information could beadded to the report including clinical history or a research studyidentification number.

As mentioned with reference to FIG. 1, the system may facilitate lesionanalysis of the same lesions over multiple timepoints to track objectivetumor response over time in a manner that improves efficiency and/orreduces read times. FIG. 28 illustrates an example of a displayinterface rendering as the system provides different sets ofcross-sectional images for a particular patient captured at differenttimepoints. In FIG. 28, the first cross-sectional image set 2802 shownon the right portion of the user interface of FIG. 28 corresponds toprevious-timepoint cross-sectional images with lesion analysis alreadyperformed on at least some of the previous-timepoint cross-sectionalimages. The second cross-sectional image set 2804 shown on the leftportion of the user interface of FIG. 28 corresponds to later-timepointcross-sectional images. In the example shown in FIG. 28, the differentsets of cross-sectional images 2802 and 2804 are displayedsimultaneously such that the user can navigate through either set bymanipulating I/O device interface(s) 106.

In some instances, the system accesses a database that includes entrieswith information associated with the previously analyzed lesions (e.g.,location information/labeling, shape/segmentation, major and/or minoraxes, pixel area, slice location, etc.) and may display one or morelists including representations of the information stored in thedatabase. For instance, the system may render a first list 2806associated with the previous-timepoint cross-sectional images (the firstset of cross-sectional images 2802) to facilitate guided presentationfunctionality similar to the functionality described hereinabove withreference to FIGS. 23-26. In some instances, the system simultaneouslypresents a second list 2808 associated with the later-timepointcross-sectional images (the second set of cross-sectional images 2804)that includes at least some of the information from the database withinformation for the previously analyzed lesions. For instance, because,in the example shown, the same lesions are analyzed over time to measureobjective tumor response, the location information/labeling associatedwith previously analyzed lesions of the previous-timepointcross-sectional images (the first set of cross-sectional images 2802)will be the same as the currently analyzed lesions in thelater-timepoint cross-sectional images (the second set ofcross-sectional images 2804). Accordingly, the location informationshown in both the first list 2806 and the second list 2808 may be thesame.

The second list 2808 shown in FIG. 28 in connection with thelater-timepoint cross-sectional images (the second set ofcross-sectional images 2804) may also be associated with a differentdatabase (or a different portion of the same database as the databasethat stores the information associated with the previously analyzedlesions) for storing information associated with the lesions asrepresented in the later-timepoint cross-sectional images, as will benow described below. Because the same lesions are analyzed over time, asnoted above, location information of the previous-timepoint lesionanalysis entries may be copied from the previous-timepoint databaseentries into the later-timepoint database entries for thelater-timepoint lesion analysis.

Certain aspects and/or features of the guided presentation describedhereinabove referring to FIGS. 23-26 may also apply in later-timepointanalysis of previously analyzed lesions, as depicted in FIGS. 28-33. Forinstance, a user may select a control (e.g., a “Start Autopilot” button2810, as shown in FIG. 28) to initiate a guided analysis of lesionsrepresented in the second set of cross-sectional images 2804 based atleast in part on information associated with the previously analyzedlesions (e.g., represented in the first list 2806). FIG. 28 illustratesa user control icon 2812 positioned over the “Start Autopilot” button2810, which may initiate a guided analysis of the lesions representedsecond list 2808 associated with the second set of cross-sectionalimages 2804 (e.g., the later-timepoint cross-sectional images), wherethe lesions represented in the second list 2808 correspond to thelesions represented in the first list 2806 (e.g., associated with thefirst set of cross-sectional images 2802).

In response to detecting user input associated with initiating a guidedanalysis of the later-timepoint lesions of the second list 2808 (whetherby selecting a dedicated button for this purpose or by selecting anentry of one of the lists), the system may utilize various components(e.g., image processing module(s) 110, machine learning module(s) 120),etc.) to identify one or more later-timepoint cross-sectional images ofthe second set of cross-sectional images 2804 that correspond to one ormore previous-timepoint cross-sectional images of the first set ofcross-sectional images 2802. Such functionality may be carried out toidentify a predicted matching cross-sectional image from the second setof cross-sectional images 2804 that matches a previous-timepointcross-sectional image of the first set of cross-sectional images 2802.Cross-sectional images from the different sets of cross-sectional imagesmay be regarded as “matching” when they both include substantiallysimilar representations of structures of the body of the patient,include any lesions represented therein.

Providing such functionality may expedite the process of locating alater-timepoint cross-sectional image (from the second set ofcross-sectional images 2804) that includes a representation of a lesionthat is also represented in (and previously analyzed according to) aprevious-timepoint cross-sectional image (from the first set ofcross-sectional images 2802). The image processing module(s) 110 mayutilize any suitable technique(s) for identifying a predicted matchingcross-sectional image, such as image co-registration and/or othermethods.

FIG. 29 illustrates an example of a display interface rendering as thesystem provides a predicted matching cross-sectional image that attemptsto find a matching target lesion that corresponds to a previouslyanalyzed target lesion. In FIG. 29, the system detected user inputinitiating the guided analysis of the second set of cross-sectionalimages 2804 based on information associated with previously analyzedlesions (represented in the first list 2806). In other instances, thesystem may initiate or continue guided analysis in response to receivinguser input for navigating to a next cross-sectional image associatedwith an analyzed lesion, or the system may receive user input directedto a particular list entry of one of the lists.

In any case, in response to the received user input, the system mayidentify a particular list entry of the first list 2806 and may displaythe previous-timepoint cross-sectional image from the first set ofcross-sectional images 2802 that is associated with the lesionrepresented in the particular list entry. For example, FIG. 29 shows thesystem responding to user input directed toward the first entry of thefirst list 2806, which is associated with the axillary target lesion 404shown and described referring to FIGS. 6-8, 14, and 24. Accordingly, thesystem may display the axillary target lesion 404 as represented in theappropriate previous-timepoint cross-sectional image 302, and the systemmay also display an overlay of the segmentation and axes associated withthe axillary target lesion 404.

FIG. 29 also shows that the system has identified a predicted matchinglater-timepoint cross-sectional image 2902 that may include a latertimepoint representation of the axillary lesion 404 previously analyzed.In the example of FIG. 29, the system automatically navigates to anddisplays slice 14 of 271 of the second set of cross-sectional images2804 (as indicated over the upper-left hand region of the left side ofthe user interface) in response to the user input described above forinitiating or continuing the guided analysis of the second set ofcross-sectional images 2804. As noted above, the system may identifyslice 14 as a predicted matching cross-sectional image 2902 by imageco-registration or another method utilizing image processing module(s)110 and/or machine learning module(s) 120.

As shown in FIG. 29, the predicted matching cross-sectional image 2902is displayed contemporaneously with the previous-timepointcross-sectional image 302 showing the previously analyzed axillarytarget lesion 404. The contemporaneous presentation of thesecross-sectional images may allow the user to rapidly assess thesimilarity between the predicted matching later-timepointcross-sectional image 2902 and the previous-timepoint cross-sectionalimage 302 to determine whether to modify the later-timepointcross-sectional image (e.g., navigate to a different, neighboring slice)before performing lesion analysis on the later-timepoint representationof the target lesion that was previously analyzed.

FIG. 30 illustrates an example of a display interface rendering as thesystem displays a matching cross-sectional image 3002 after receivinguser input modifying or navigating away from the initial predictedmatching cross-sectional image 2902. In FIG. 30, the user has provideduser input modifying or navigating away from the predicted matchinglater-timepoint cross-sectional image 2902 by navigating to slice 15 ofthe second set of cross-sectional images 2804 rather than slice 14 ofthe second set of cross-sectional images 2804. Even with thisuser-directed change, the automated prediction of a matchingcross-sectional image from the second set of cross-sectional images 2804may bring the user within close range to an optimal matchinglater-timepoint cross-sectional image, so the disclosed systems mayimprove read times even without always predicting the best matchingcross-sectional image.

With the matching later-timepoint cross-sectional image selected, thesystem may facilitate lesion analysis on the later-timepointrepresentation of the lesion that corresponds to the previously analyzedlesion. As shown in FIG. 30, a user control icon 3004 is positioned overa matching lesion 3006 in the later-timepoint matching cross-sectionalimage 3002 that corresponds to the axillary target lesion 404 previouslyanalyzed and shown in the displayed previous-timepoint cross-sectionalimage 302. The user may thereafter select a pixel or pixel patch withinthe later-timepoint lesion (e.g., the matching lesion 3006) to beginlesion analysis thereon.

FIG. 31 illustrates an example of a display interface rendering as thesystem determines shape and location information for the matching targetlesion 3006. The system may implement any of the operations, functions,and/or processes described hereinabove with reference to FIGS. 1-22 todetermine shape and/or location information associated with the matchinglesion 3006. For instance, the system may utilize image processingmodule(s) 110 and/or machine learning module(s) 120 to determine apredicted shape 3102 or segmentation associated with the matching lesion3006. The user may also modify the segmentation as described above. Thesystem may associate a matching lesion marker 3104 (e.g., implemented asa simple “X” or other symbol for non-target lesions or other findings,or a major axis and/or a minor axis rendering for target lesions) withthe matching lesion 3006 and store the matching lesion marker 3104within the database associated with the matching lesion 3006 and/or thecross-sectional image 3002. The segmentation and/or other informationfor the matching lesion 3006 may also be stored. Furthermore, the systemmay display lesion marker, predicted shape, and/or other informationassociated with the matching lesion 3006 and the previously analyzedlesion 404 simultaneously within and/or overlaid on their respectivecross-sectional images (3002 and 302, respectively).

It should be noted that the foregoing functionality associated with theguided analysis of lesions represented in later-timepoint second set ofcross-sectional images 2804 may be applied to target lesions (asdescribed in FIGS. 29-31), non-target lesions, and/or otherlesions/findings. For instance, FIG. 32 illustrates an example of adisplay interface rendering as the system displays a matchingcross-sectional image 3202 that matches a prior-timepointcross-sectional image 2002 that includes a representation of apreviously analyzed non-target lesion 2008 (e.g., as described withreference to FIG. 20). The previously analyzed non-target lesion 2008corresponds to the first non-target lesion entry of the first list 2806.The system determines and/or displays a lesion marker 3206 for amatching non-target lesion 3204 of the matching cross-sectional image3202 that corresponds to the previously analyzed non-target lesion 2008.The lesion marker 3206 may be determined at least partially based onuser input (e.g., input directed toward a pixel region in the matchingcross-sectional image 3202.

In some instances, for non-target lesions, the system may refrain fromobtaining segmentation and/or axis information associated with thematching non-target lesion 3204. For instance, as shown in FIG. 32, thesystem may prompt the user to qualitatively analyze the matchingnon-target lesion 3204 and rapidly provide input identifying whether thematching non-target lesion 3204, as compared with the previouslyanalyzed non-target lesion 2008, is absent or not, present orpathologic, has exhibited unequivocal progression, or was not evaluated.To facilitate rapid analysis/input, the system may provide buttons 3208corresponding to the non-target lesion classifications noted above,including a “Absent/Not Pathologic” button, “Present/Pathologic” button,an “Unequivocal Progression” button, or a “Not Evaluated” button. Suchuser input for non-target lesions may also become stored in a databaseassociated with analysis for later-timepoint lesions.

Similarly, FIG. 33 illustrates an example of a display interfacerendering as the system displays a matching cross-sectional image 3302that matches a prior-timepoint cross-sectional image 2102 that includesa representation of a previously analyzed lesion 2104 that is not atarget or non-target lesion (as described with reference to FIGS. 21 and22). The previously analyzed lesion 2104 is noted in the portion of thefirst list 2806 dedicated to “Other Findings.” The system determinesand/or displays a lesion marker 3306 for a matching lesion 3304 that isnot a target lesion or a non-target lesion and that corresponds to apreviously analyzed lesion 2104 that was not a target lesion or anon-target lesion. The lesion marker 3306 may be determined at leastpartially based on user input (e.g., input directed toward a pixelregion in the matching cross-sectional image 3302).

Upon receiving user input identifying the location of the lesion in thepredicted matching later-timepoint cross-sectional image, the system mayprompt the user to indicate whether the lesion is “Resolved,”“Improved,” “Unchanged,” “Worsened,” or “Not Evaluated” via one or morebuttons 3308. Such user input for the lesion 3304 may also become storedin a database associated with analysis for the matching lesion 3304.

As indicated hereinabove, the system may provide functionality fororganizing and/or compiling the information stored for one or morelesions at one or more timepoints to generate oncological and/or patientreports. FIG. 34 illustrates an example of a display interface renderingas the system provides a summary report 3402 based on multiple lesionsanalyzed at different timepoints. The summary report 3402 may indicatethe criteria under which the summary report 3402 was generated (e.g.,“Freeform RECIST”), a generalization of the treatment response based onthe lesion metrics determined for the analyzed lesions (“StableDisease”), as well as various charts and/or tables indicating lesionmetrics for lesions analyzed over one or more different timepoints.

For instance, the summary report 3402 provides a listing of targetlesions 3404 that includes information associated with the targetlesions analyzed over multiple timepoints. For example, the listing oftarget lesions 3404 of the summary report 3402 includes indications ofthe various anatomical locations of the analyzed target lesions as wellas indications of the most recently measured major and minor axis lengthfor the analyzed target lesions. As shown in FIG. 34, the listing oftarget lesions 3404 also includes a summary that provides useful metricsfor the analyzed target lesions, such as the current sum of the axislengths of the analyzed target lesions (which may include major axes,minor axes, or a combination) and/or changes in the sum of axis lengthsover various timepoints (e.g., change relative to the baseline ormeasurements associated with a first timepoint, changes relative to thelowest sum found, changes relative to a sum associated with a priortimepoint, etc.).

The summary report 3402 of FIG. 34 also includes a listing of non-targetlesions 3406 that may include lesion location information for analyzednon-target lesions and/or determined response for analyzed lesions, suchas whether the non-target lesion is present/pathologic,absent/non-pathologic, is unequivocally progressing, and/or notevaluated (e.g., according to user input provided under longitudinalanalysis of non-target lesions over multiple timepoints, as shown inFIG. 32).

The summary report 3402 of FIG. 34 also includes a listing of otherfindings 3408, which may include lesions or other findings that weretracked during analysis of cross-sectional images for a patient (e.g.,as shown in FIGS. 21-22 and 33). Such lesions may comprise items ofinterest found that do not squarely fit within a tumor responsecriterion, but that reviewers believe are relevant to the wellbeing ofthe patient. The listing of other findings 3408 may include identifyinginformation for the other findings provided during analysis of the otherfindings, as well as further action that should be taken.

FIG. 34 also illustrates that the summary report 3402 may comprise agraphical representation 3410 illustrating the composite length of theanalyzed target lesions according to analysis performed at differenttimepoints. The composite length may comprise a summation of variouslengths of various analyzed target lesions. As noted above, a targetlesion may have a minor axis and a major axis associated therewith, andthe system may select from among the minor and major axes forcontribution to the composite length depending on the tumor responsecriteria (e.g., under some tumor response criteria, the system mayselect minor axes for lymph nodes, and major axes for masses). Thegraphical representation 3410 may provide users with an intuitiverepresentation of tumor progression and/or treatment response.

The summary report 3402 may also comprise other information, such aswhether new sites of disease have been identified (e.g., in a mostrecently performed analysis of cross-sectional images for a patient), aswell as a report information section 3412. New sites of disease could becharacterized as possible or definite. The report information section3412 may comprise various types of information related to the report,such as the date the report was generated and/or an intended audience orstatus of the report, etc. As noted above, any number of specialists maycontribute lesion analysis for cross-sectional images associated with asingle patient (e.g., for any number of timepoints). Thus, the reportinformation section 3412 of a summary report 3402 may compriseinformation for one or more readers involved in analyzing the lesionsrepresented in the summary report 3402.

One will understand that the particular layout and/or form of the datarepresented in the summary report 3402 may be varied in differentimplementations, and that a summary report 3402 may comprise anyadditional or alternative information.

The summary report 3402 may emphasize certain data based on the tumorresponse criteria and/or other factors associated with the report. Forexample, in the listing of target lesions 3404, the minor or short axesare highlighted based on the types of target lesions measured (e.g.,lymph nodes) in accordance with the criterion used (e.g., FreeformRECIST).

Other types and/or forms of reports are within the scope of thisdisclosure. For example, FIG. 35 illustrates a detailed table report3502 based on one or more lesions analyzed at different timepoints andprovides detailed lesion information for lesions measured at eachtimepoint. For instance, whereas the summary report 3402 of FIG. 34 onlyshowed axis length for each target lesion and status classifications foreach non-target lesion for the most recently analyzed timepoint, thedetailed table report 3502 may show axis length and statusclassifications for lesions associated with more than one or allanalyzed timepoints.

By way of illustration, the target lesion section 3504 of the detailedtable report 3502 of FIG. 35 includes information for target lesionsanalyzed at two different timepoints, such as major and minor axislength, target lesion location and type (e.g., whether lymph node ormass), respective target lesion length sums for each differenttimepoint, and analysis appertaining thereto (e.g., indicating whether atimepoint provides a baseline sum or a lowest sum, indicating changerelative to a baseline sum or a lowest sum, change relative to a priortimepoint, etc.). Furthermore, the non-target lesion section 3506 of thedetailed table report 3502 of FIG. 35 may include information for allnon-target lesions analyzed over the different timepoints. Also,although not explicitly shown, the detailed table report 3502 maycomprise additional details associated with which reader performedanalysis on which lesions and may further comprise a section for otherfindings.

FIG. 36 illustrates an example of a key image report 3602 based on oneor more lesions analyzed at different timepoints. The key images 3604 ofthe key image report 3602 may comprise, as shown in FIG. 36, zoomedviews of the analyzed lesions as represented in their respectivecross-sectional images. The key images 3604 may include indicationsand/or highlighting of shape/segmentation and/or lesion markers for theanalyzed lesions. As shown, the key images 3604 may be arranged based ontimepoint to illustrate progression of the lesions over time (e.g., withtreatment), and location information for any lesion may be included in akey image report. For example, different key images 3604 of the samelesion associated with different timepoints may be arranged adjacentlyto intuitively illustrate progression of the lesion over time forpatients and/or practitioners.

Any of the foregoing reports described referring to FIGS. 27 and 34-36may be used for oncological review, for presentation to a patient,and/or for other purposes. Those skilled in the art will recognize thatthe form and content of the reports, as well as the particular style andformatting of the user interface renderings shown herein, are providedas examples only and do not limit the scope of the presently disclosedembodiments. For example, the elements of the reports described hereinare not exclusive to any particular type of report, and any element ofany report may be combined with any other element(s) of any otherreports to form a report in accordance with the present disclosure.

A set of cross-sectional medical images associated with a particularpatient may include cross-sectional images for various parts of thepatient's body. For instance, the set may include scans of the user'shead, neck, chest, and/or abdomen. In many instances, a physician orother reviewer of cross-sectional images for lesion analysis is onlyspecialized, trained, and/or authorized to analyze certain types ofcross-sectional images. For example, one reviewer may only specialize inreviewing head and neck cross-sectional images, whereas another reviewermay only specialize in reviewing abdominal cross-sectional images. Somereviewers may be authorized to review all cross-sectional images,regardless of the anatomical location represented in the cross-sectionalimages.

In instances where a reviewer will only review a subset ofcross-sectional images within a set of cross-sectional images (e.g.,because of the radiology subspecialty of the reviewer), presenting theentire set of cross-sectional images to the reviewer may slow theanalysis speed of the reviewer and may give rise to unintentionalannotations being implemented into cross-sectional images that areoutside of the reviewer's expertise.

Additionally, not all radiology specialists may be simultaneouslyavailable to review a set of cross-sectional images that includes imagesfor review by different radiology specialists. Therefore, problems mayarise when a patient report is generated prematurely before allcross-sectional images within a set of cross-sectional images associatedwith the patient have been processed by appropriate specialists forlesion analysis. For instance, reports generated under suchcircumstances may omit target lesions that went unanalyzed by an absentspecialist, and the omission of target lesions may skew tumor responseresults.

Furthermore, it is uncommon for different radiology specialists tocommunicate with one another when interpreting exams for the samepatient. This separation of reporting often gives rise to problems whenthe patient has a condition that can affect different body regions, suchas advanced cancer. Accordingly, the reports from different radiologicspecialists may not agree, may have discrepancies, or may haveconflicting information with one another when considering apatient-level review by a treating clinician. It is uncommon forradiology specialists to notice these problems when working separatelyfrom one another (e.g., when reviewing at different times and/or fromdifferent places).

Because many conditions that affect different regions are treated at thepatient level, the aforementioned temporal and/or spatial separationbetween different radiology specialists when reviewing a patient's examsoften gives rise to problem under existing techniques. For example,systemic cancer therapies are used to treat metastatic disease orlymphoma that separately involves the neck, chest, abdomen, and pelvis.In such examples, a single patient-level view of the response may bemore important and/or beneficial than multiple separate reports abouteach separate body region (e.g., a separate neck report, chest report,abdomen, etc.). Furthermore, the reporting styles of differentsubspecialty radiologists could lead to additional errors. For instance,a slight increase in size in two different body regions could beinterpreted differently by different reviewing physicians, with oneradiologist reporting progressive disease and with another radiologistreporting stable disease. This conflicting information is difficult toresolve at the patient level and is confusing for both treatmentproviders and patients.

Accordingly, in some embodiments, the presently disclosed systems andmethods for lesion analysis include techniques for managing access tocross-sectional images within a set of cross-sectional images and forcontrolling the generation of a patient report, which may comprise acomposite report that includes analysis from multiple specialists. Thecomposite report could be a patient-level report.

Radiology subspecialties may include, for example, neuroradiology orbody radiology (which can include cardiothoracic, chest, and/or abdomensubspecialties) or other subspecialties as known in the art. It will beappreciated that certain radiologists may be proficient at reviewingcross-sectional images under multiple or all radiologicalsubspecialties.

The radiology specialties associated with a particular user (e.g., aphysician reviewer) may be indicated by a user profile associated withthe particular user. As used herein, a “user profile” refers broadly toany computer-readable/interpretable information that indicatesattributes, characteristics, settings, information, and/or the likeassociated with a particular user. A user profile may indicate one ormore radiology specialties or subspecialties associated with a useroperating a system for lesion analysis. In some instances, a userprofile for a particular user may persist in computer memory outside ofa particular lesion analysis session (e.g., to be used in future lesionanalysis sessions for the particular user). For example, a user profilemay be selectively modified (e.g. by an administrator or by the user)outside of lesion analysis sessions to enable different radiologysubspecialties or combinations thereof. In some instances, a userprofile for a particular user may be specific to a particular lesionanalysis session (e.g., the user profile may be established based onuser input provided by one or more users at the beginning of a lesionanalysis session).

In accordance with the present disclosure, a system may be configured toprovide subsets of cross-sectional medical images to users for analysisbased on a user profile associated with the user (e.g., based onradiology subspecialties associated with the users, as indicated by theuser profiles).

To facilitate such functionality, in some instances, a set ofcross-sectional images (and/or one or more individual cross-sectionalimages thereof) may comprise or be associated with metadata, which mayindicate one or more body regions or radiologyspecialties/subspecialties associated with one or more subsets ofcross-sectional images thereof. As used herein, metadata refers broadlyto any computer-readable information that provides any information aboutany other computer-readable information (e.g., cross-sectional imagesand/or sets/subsets thereof).

Various techniques may be employed for associating metadata withcross-sectional images and/or sets thereof (e.g., to indicate one ormore body regions and/or radiology specialties/subspecialties associatedtherewith). For example, metadata may be associated with thecross-sectional images or set thereof upon image capture (e.g., by theradiologic device 104) or soon thereafter (e.g., by an entity separatefrom the radiologic device 104).

In some instances, a system prompts a user to associate cross-sectionalimages of a set of cross-sectional images with different body regionsand/or radiology specialties/subspecialties. FIG. 37 illustrates aconceptual representation of prompting a user to associatecross-sectional images of a set of cross-sectional images with differentspecialties. For example, FIG. 37 illustrates a system 3700, which mayat least partially correspond to a computing system 100 and/or acomputing device 130 a, 130 b, 130 c, or other component(s) describedhereinabove with reference to FIG. 1. The system 3700 may comprise or bein communication with a user device 3702, which may at least partiallycorrespond to a computing system 100 and/or a computing device 130 a,130 b, 130 c, described hereinabove with reference to FIG. 1. A user(e.g., a physician reviewer) may operate the user device 3702, and theuser may be associated with a user profile 3704. As noted above, theuser profile may indicate one or more radiology specialties and/orsubspecialties associated with the user.

The system 3700 may receive a request, in association with the userprofile 3704, to review a set of cross-sectional images 3720 that isaccessible to the system 3700. The set of cross-sectional images 3720may comprise different cross-sectional images capturing differentportions of a patient's body and therefore being appropriate forrevision by radiologists associated with differentspecialties/subspecialties. In some instances, the set ofcross-sectional images 3720 is not already associated with metadataindicating appropriate specialties/subspecialties for the variouscross-sectional images of the set. Thus, in some implementations, inresponse to receiving the request in association with the user profile3704, the system provides the set of cross-sectional images 3720 to theuser device 3702 and prompts a user operating the user device 3702 toidentify appropriate specialties/subspecialties for the variouscross-sectional images of the set of cross-sectional images 3720 (asindicated in FIG. 37 by the arrow extending from the set ofcross-sectional images 3720 toward the user device 3702). In someinstances, the system 3700 requires the user to complete theidentification/classification of cross-sectional images prior topermitting the user to review any portion of the set of cross-sectionalimages 3720.

The user operating the user device 3702 may provide input 3706 inaccordance with the prompt described above, which may indicate differentspecialties/subspecialties for different subsets of the set ofcross-sectional images. By way of illustration, FIG. 37 depicts that theinput 3706 indicates that a first subset 3708 of the set ofcross-sectional images 3720 is associated with a head and neck bodyregion or specialty/subspecialty, a second subset 3710 of the set ofcross-sectional images 3720 is associated with a chest body region orspecialty/subspecialty, and a third subset 3712 of the set ofcross-sectional images 3720 is associated with an abdomen body region orspecialty/subspecialty. Based on the user input, the system mayassociate metadata 3722 with the various subsets 3708, 3710, 3712 ofcross-sectional images and/or the set of cross-sectional images 3720indicating the appropriate body regions or specialties/subspecialtiesfor the various subsets 3708, 3710, 3712 of cross-sectional images.

Based on the metadata (however obtained/established), a system may beprepared to provide subsets of cross-sectional images to different usersfor analysis based on specialties/subspecialties associated with theusers. For instance, FIG. 38 illustrates a conceptual representation ofidentifying and providing different subsets of cross-sectional imagesfor different users based on specialties associated with the users. Inparticular, FIG. 38 shows a system 3800 (which may correspond to thesystem 3700 described above) that receives requests to analyze at leasta portion of the set of cross-sectional images 3720 in association withuser profiles 3804A, 3804B, and 3804C that are associated with usersoperating user devices 3802A, 3802B, and 3802C. The user devices 3802A,3802B, and 3802C may at least partially correspond to the user device3702 described above, and it will be appreciated, in view of the presentdisclosure, that the user devices 3802A, 3802B, and 3802C may representseparate physical devices or may represent one or more of the samephysical devices operated at different times (e.g., by different usersassociated with different user profiles).

As noted above, the user profiles 3804A, 3804B, and 3804C may indicatedifferent radiology specialties/subspecialties for different usersoperating the different user devices 3802A, 3802B, and 3802C. Forexample, user profile 3804A may indicate a head and neck specialty, userprofile 3804B may indicate a chest specialty, and 3804C may indicate anabdomen specialty.

In some instances, in response to the received requests, the system mayidentify the respective user profiles 3804A, 3804B, and 3804C and thespecialties/subspecialties indicated by the user profiles 3804A, 3804B,and 3804C (e.g., head and neck, chest, and abdomen, respectively). Thesystem may then identify, based on the user profiles 3804A, 3804C, and3804B, appropriate respective subsets of cross-sectional images from theset of cross-sectional images 3720 to provide to the various userdevices 3802A, 3802B, and 3802C. For example, the system may provide thefirst subset 3708 comprising head and neck cross-sectional images touser device 3802A based on user profile 3804A, the system may providethe second subset 3710 comprising chest cross-sectional images to userdevice 3802B based on user profile 3804B, and the system may provide thethird subset 3712 comprising abdomen cross-sectional images to userdevice 3802C based on user profile 3804C (indicated in FIG. 38 by thearrows extending from the various subsets toward the various userdevices). For a particular user device/user profile, the system 3800 mayrefrain from providing subsets of cross-sectional images that are notrelated to the same specialty/subspecialty indicated by the userprofile, which may advantageously allow the user to focus on thecross-sectional images within their radiological specialty/subspecialty.

Upon receiving an applicable subset of cross-sectional images 3708,3710, and/or 3712, users operating the various user devices 3802A,3802B, and 3802C may carry out lesion analysis according to theprinciples and/or techniques described herein (which may be at leastpartially synchronous or asynchronous in time, depending on theavailability of the various users). For example, the user devices 3802A,3802B, and 3802C may receive user input that marks, segments, and/orlabels one or more lesions represented within the corresponding subset3708, 3710, or 3712 of cross-sectional images. In some instances, theuser devices 3802A, 3802B, and 3802C may operate to provide guidedpresentations of respective selections of the respective subsets ofcross-sectional images that include analyzed lesions (e.g., a lesionthat was previously marked and/or segmented, see FIGS. 23-26 andattendant description). In some instances, the user devices 3802A,3802B, and 3802C may operate to facilitate longitudinal analysis oflesions (e.g., see FIGS. 28-33 and attendant description)

Lesion information (e.g., based at least in part on user input at thevarious user devices 3802A, 3802B, and 3802C) associated with variouslesions represented in the various subsets 3708, 3710, and 3712 maybecome organized into a subspecialty report. For example, FIG. 39 showssubspecialty reports 3902A, 3902B, and 3902C based on user inputprovided at the various respective user devices 3802A, 3802B, and 3802C.The subspecialty report 3902A may include lesion information for lesionsrepresented within the first subset 3708 of cross-sectional images andanalyzed using user device 3802A. Similarly, subspecialty report 3902Bmay include lesion information for lesions represented within the secondsubset 3710 of cross-sectional images and analyzed using user device3802B, and subspecialty report 3902C may include lesion information forlesions represented within the third subset 3712 of cross-sectionalimages and analyzed using user device 3802C.

As used herein, a “subspecialty report” may refer to an at leastpartially formalized or organized report similar to one or more of thosedescribed herein with reference to FIGS. 27 and 34-36, or may refer moregenerally to any lesion information stored based at least in part onuser input (e.g., from a radiologist reviewing cross-sectional images)provided at a user device (e.g., one of the various user devices 3802A,3802B, and 3802C).

As depicted in FIG. 39, a system 3900 may receive the varioussubspecialty reports 3902A, 3902B, and 3902C. The system 3900 may atleast partially correspond to the system 3800 described above. Thesystem 3900 may be configured/configurable to generate a compositereport 3904 based on the various subspecialty reports 3902A, 3902B, and3902C received. However, FIG. 39 shows that, in some instances, thesystem 3900 is configured to refrain from generating or finalizing acomposite report 3904 (e.g., for sending to an oncologist, patient, orother end user of the report) until the system 3900 verifies that allappropriate subspecialty reports 3902A, 3902B, and 3902C have beenreceived.

For example, FIG. 39 illustrates a decision block 3906 associated withthe system, whereby the system may determine whether all expectedsubspecialty reports are present. For instance, continuing with theabove example where the set of cross-sectional images 3720 is associatedwith three separate subsets 3708, 3710, and 3712 of cross-sectionalimages corresponding to different specialties/subspecialties, the system3900 may be configured to refrain from generating or finalizing acomposite report 3904 until a subspecialty report (comprising lesioninformation obtained from lesion analysis performed) has been receivedin association with each of the different subsets 3708, 3710, and 3712of the set of cross-sectional images 3720. For example, as discussedhereinabove, different radiology specialists may perform lesion analysisat different times, even when analyzing different subsets ofcross-sectional images for a single patient (e.g., with one specialistanalyzing abdomen images, with another specialist analyzing chestimages, etc.). Thus, different subspecialty reports may be created oravailable at different times.

When the system 3900 determines that all applicable subspecialty reportshave been received, the system 3900 may proceed to generate, finalize,update, or send a composite report 3904 for use by an end user (e.g., anoncologist or patient). For example, a system 3900 may be configured tocreate or update the composite report 3904 in response to determiningthat a final reader has completed their evaluation of a patient, or inresponse to determining that each subset of images that is queued orintended for analysis for a particular patient has been analyzed by oneor more radiologists. Such functionality may advantageously preventpatient or oncological reports from being prematurely generated based onincomplete information (e.g., in situations where not all specialistsare available to perform lesion analysis within the same time period),thereby improving patient care.

It should be appreciated that the composite report can be generated froma subset of subspecialty reports, and the inclusion of any number ortype of subspecialty reports within a composite report can be selectedby the physician or other healthcare provider. For example, asubspecialty radiologist can perform their respective analysis of thecross-sectional images and generate a report that includes only thosefindings related to their analysis and/or subspecialty. Such reports canbe formatted as a patient-level report having, for example, a graph,table, key images, and/or structured text representing information fromthe given subspecialty radiologist.

In addition to indicating a radiology subspecialty for which the user isenabled to perform lesion analysis on cross-sectional images, userprofiles may allow for a personalized experience for users whenperforming lesion analysis on cross-sectional images. Providing apersonalized experience may improve user attentiveness and/or accuracywhen performing lesion analysis as described herein.

In this regard, a user profile may indicate one or more systeminteraction preferences for the user. The interaction preferences mayinclude one or more interaction presentations, such as sounds, images,animations, and/or combinations thereof.

As described hereinabove, the system may provide various controlmechanisms (e.g., via I/O device interface(s) 106) to allow a user tocontrol certain aspects of lesion analysis. These controls may include,for example, selecting a position within a pixel region corresponding toa lesion represented in a cross-sectional medical image,tracing/segmenting a pixel region associated with a lesion, selectinglocation information for a lesion, navigating through a guidedpresentation of a subset of cross-sectional medical images that includeone or more marked lesions, selecting a representation of a list entryassociated with a cross-sectional medical image that includes one ormore marked lesions, accepting a predicted lesion shape or lesionlocation information generated by a machine learning module, rejecting apredicted lesion shape or lesion location information generated by amachine learning module, triggering display of a report comprisinginformation associated with one or more marked lesions present withinthe plurality of cross-sectional medical images, navigating through aset of cross-sectional images, and zooming a display of across-sectional image.

Accordingly, the system may identify a user profile of the useraccessing the system and associate one or more controls provided by thesystem with one or more interaction preferences/presentations specifiedwithin the user profile. In some implementations, the interactionpreferences include a different interaction presentation for each of thecontrols provided by the system. Subsequently, in response to detectinguser input operating one of the associated controls, the system presentsthe applicable interaction presentation.

By way of non-limiting example, a user's profile may indicate aninteraction preference that includes presenting an image of acalligraphy pen as a mouse cursor instead of a traditional cursor. Thesystem may associate the interaction presentation of presenting thecalligraphy pen with the control of tracing a pixel region associatedwith a lesion. Thereafter, when the user's profile is activated and inresponse to detecting user input for tracing a pixel region associatedwith a lesion, the system replaces the mouse cursor with the image ofthe calligraphy pen, and the image persists while the user traces thepixel region.

In some embodiments, the user profile may be associated withuser-selected sound and/or visual effects profiles for pre-selectedactions. For example, selecting target lesions or non-target lesions maybe associated with a first sound while segmenting or correctingauto-segmentation results can be associated with a different sound, suchas a swooshing sound of a katana or humming of a saber as portions ofthe segmented lesion are lopped off by the re-segmentation action. Thesounds may be preset or configurable by the user. Additionally, oralternatively, the sounds may be thematic in accordance with a genre(e.g., Western, Science Fiction, Steam Punk, Medieval, etc.) or specificto a movie, video game, song, and/or other popular culture creation(e.g., television show, play, cartoon, etc.). A visual effect couldmimic lights, colors and/or other aspects of the same sound effectstheme and could mimic visual effects from a specific movie, video game,song or other popular culture creation. The sounds and visual effectscould be mapped to specific keys or actions in the user interface. Thesound and visual effects could be used to improve user attention,enjoyment, engagement, and/or efficiency and could be used in productdemonstrations, product trial versions, product challenges, and finalproduction versions of the user interface. The sound and visual effectscould be selectively turned off by the user, and the sound effects couldinclude background music. Background music could follow the same themeas the sound and visual effects or could diverge from these at thepreference of the user.

In addition to defining interaction presentations that may be associatedwith system controls, a user profile may also include a user interfacetheme that alters a rendering of at least some of the controls or theuser interface for lesion analysis as described and shown herein.

In some embodiments, the user interface and associated system track thetime and/or number of errors made by a user and ranks the user'sperformance. Points can be attributed to various actions and gained orlost in accordance with the user's performance. In some embodiments, theuser interface displays the gain or loss of points in real time and/orafter a task or session is completed. For example, the user interfacemay display a target lesion selected by the user concomitantly with apoint value in accordance with the selection's adherence to one or morerules associated with the tumor response criterion/criteria followedduring the reading and/or in accordance with a proper identification. Inthis way, different user metrics can be tracked over time and comparedto other users to identify weaknesses and/or strengths of various users.In some instances, an attending radiologist can utilize such a system toscore various residents who may have initially read the images. In someembodiments, such a review can be undertaken using the guidedpresentation functionality discussed above with respect to FIGS. 23-26and/or the longitudinal analysis functionality discussed above withrespect to FIGS. 28-33.

Example Methods Associated with Lesion Analysis

The following discussion now refers to a number of methods and methodacts that may be performed. Although the method acts may be discussed ina certain order or illustrated in a flow chart as occurring in aparticular order, no particular ordering is required unless specificallystated, or required because an act is dependent on another act beingcompleted prior to the act being performed.

FIGS. 40, 41, 42 and 43 illustrate example flow diagrams 4000, 4100,4200 and 4300, respectively, which depict acts associated with analyzinglesions in cross-sectional medical images. The discussion of the variousacts represented in the flow diagrams may include references to varioushardware components described in more detail with reference to FIGS. 1,17, 38, and/or 39. One will appreciate, in view of the presentdisclosure, that various embodiments may omit one or more of the actsdescribed hereinbelow.

Act 4002 of flow diagram 4000 of FIG. 40 includes presenting across-sectional medical image to a user on a display. Act 4002 may becarried out using one or more components (e.g., I/O device interface(s)106, hardware processor(s) 108, image processing module(s) 110, hardwarestorage device(s) 112, data processing module(s) 114, primary database116, machine learning module(s) 120, and/or others) of a computingsystem 100 or similar device. The cross-sectional medical image maycomprise one or more of CT images, CTP images, PET images, SPECT images,MRI images, or ultrasound images, and/or others. Such images may beobtained by a radiologic device 104.

Act 4004 of flow diagram 4000 includes receiving user input directed toa pixel region corresponding to a lesion represented in thecross-sectional medical image. Act 4004 may be carried out using one ormore components (e.g., I/O device interface(s) 106, hardwareprocessor(s) 108, hardware storage device(s) 112, and/or others) of acomputing system 100 or similar device. In some implementations, theuser input may comprise user input provided by a mouse cursor navigatedover the lesion represented in the cross-sectional medical image, or asimilar type of user input (e.g., provided via a touch screen).

In some instances, the presentation of the cross-sectional medical imagemay be zoomed toward the lesion represented in the cross-sectionalmedical image in response to the user input directed at the pixel regioncorresponding to the lesion.

Act 4006 of flow diagram 4000 includes identifying a predicted shape ofthe lesion represented in the cross-sectional medical image. Act 4006may be carried out using one or more components (e.g., hardwareprocessor(s) 108, image processing module(s) 110, hardware storagedevice(s) 112, data processing module(s) 114, primary database 116,machine learning module(s) 120, and/or others) of a computing system 100or similar device. In some instances, the predicted shape is identifiedbased on contrast between the pixel region corresponding to the lesionand a surrounding pixel region.

Furthermore, in some implementations, identifying the predicted shape isfurther based on differences in contrast between one or more separatepixel regions in one or more neighboring cross-sectional medical images(e.g., image slices ordinally adjacent to the cross-sectional medicalimage within which the pixel region was selected) that correspond to thepixel region to which the user input was directed and one or moreseparate surrounding pixel regions that surround the separate pixelregions in the one or more neighboring cross-sectional medical images.

In some implementations, after identifying the predicted shape, act 4006may further include presenting a rendering of the predicted shape of thelesion overlaid on the lesion represented in the cross-sectional imageto the user. In some instances, in addition to the predicted shape, apredicted major axis and a predicted minor axis of the lesion may alsobe presented, and the predicted major axis and the predicted minor axisof the lesion may be determined based at least in part on the predictedshape.

Act 4006 may further include prompting the user to accept or reject thepredicted shape as displayed to the user. User input may be receivedthat rejects the predicted shape of the lesion. User input (the same orseparate user input) may be received that modifies the predicted shapeof the lesion. For instance, a user-directed trace tool may be utilizedto modify the predicted shape of the lesion.

Act 4008 of flow diagram 4000 includes automatically determininglocation information for the lesion based on the predicted shape of thelesion. Act 4008 may be carried out using one or more components (e.g.,hardware processor(s) 108, image processing module(s) 110, hardwarestorage device(s) 112, data processing module(s) 114, primary database116, machine learning module(s) 120, and/or others) of a computingsystem 100 or similar device. The location information may compriseindication of an anatomical location of the lesion which may include,for example, an indication of whether the lesion is a mass or a lymphnode and/or an indication that the lesion is located in one of: a neck,a chest, an abdomen, a pelvis, an arm, or a leg.

In some instances, the location information for the lesion is determinedat least in part by providing the predicted shape of the lesion as aninput to a machine learning module 120 trained to identify lesionlocation information based on input shape. In some instances, thepredicted shape as modified by the user input that modifies thepredicted shape may be used as an input to the machine learning module120 to determine the location information for the lesion. The machinelearning module 120 may identify the location information for the lesionbased on the predicted shape.

The machine learning module 120 may utilize one or more additional oralternative inputs for determining the location information for thelesion, such as: metadata associated with the cross-sectional medicalimage indicating an anatomical location of the lesion, other structuresrepresented in the cross-sectional medical image, a pixel coordinate ofthe user input directed to the pixel region, and/or others.

In some instances, Act 4008 includes presenting a rendering of thelocation information to the user, which may, in some implementations, bepresented contemporaneously with a presentation of a rendering of thelesion represented in the cross-sectional medical image. Act 4008 mayalso include prompting the user to either accept the locationinformation or modify the location information. In some instances, userinput modifying the location information may be received. Such userinput may modify the location information selected/identified via themachine learning module 120 and/or may add additional locationinformation to the location information selected/identified via themachine learning module 120. For instance, such user input may include auser indication of whether the lesion is a mass or a lymph node.

In some instances, in response to receiving user input modifying thelocation information selected/identified via the machine learning module120, the machine learning module 120 automatically redetermines thelocation information by providing the predicted shape and at least aportion of the location information as modified by the user as input tothe machine learning module 120. The machine learning module may thenreidentify the location information for the lesion.

Act 4010 of flow diagram 4000 includes associating the locationinformation for the lesion with the lesion represented in thecross-sectional medical image or with the cross-sectional medical image.Act 4010 may be carried out using one or more components (e.g., I/Odevice interface(s) 106, hardware processor(s) 108, hardware storagedevice(s) 112, data processing module(s) 114, primary database 116,machine learning module(s) 120, and/or others) of a computing system 100or similar device.

Act 4012 of flow diagram 4000 includes storing the predicted shape andthe location information associated with the lesion represented in thecross-sectional medical image within a data structure. Act 4012 may becarried out using one or more components (e.g., I/O device interface(s)106, hardware processor(s) 108, image processing module(s) 110, hardwarestorage device(s) 112, data processing module(s) 114, primary database116, machine learning module(s) 120, and/or others) of a computingsystem 100 or similar device. In some instances, additional oralternative information associated with the lesion may be stored. Forinstance, a major axis and a minor axis of the lesion (e.g., determinedbased on the predicted shape) may be stored in the data structure inassociation with the lesion and in accordance with act 4012. In someinstances, the location information as modified by a user according toact 4008 becomes stored within the data structure in association withthe lesion.

Flow diagram 4000 may comprise acts that are performed using lesioninformation stored within the data structure, such as facilitating aguided presentation or review of the lesion(s) for which associatedinformation is stored within the data structure in accordance with act4012.

For example, act 4014 of flow diagram 4000 includes presenting one ormore representations of lesion information (e.g., predicted shapes,location information, major/minor axes, and/or other lesionmetrics/information) stored within the data structure to a user. Act4016 of flow diagram 4000 includes receiving user input selecting aparticular representation of the one or more representations of lesioninformation. Act 4018 of flow diagram 4000 includes rendering andpresenting the predicted shape or a major axis or a minor axis derivedfrom the predicted shape overlaid on the lesion represented in aparticular cross-sectional medical image corresponding to the particularrepresentation. Acts 4014, 4016, and/or 4018 may be carried out usingone or more components (e.g., I/O device interface(s) 106, hardwareprocessor(s) 108, image processing module(s) 110, hardware storagedevice(s) 112, data processing module(s) 114, primary database 116,machine learning module(s) 120, and/or others) of a computing system 100or similar device.

Act 4102 of flow diagram 4100 of FIG. 41 includes identifying a userprofile associated with a user accessing a system. Act 4102 may becarried out using one or more components (e.g., I/O device interface(s)106, hardware processor(s) 108, hardware storage device(s) 112, dataprocessing module(s) 114, primary database 116, machine learningmodule(s) 120, and/or others) of a computing system 100 or similardevice. The user profile may indicate a radiology specialty associatedwith the user. The radiology specialty may comprise, for example, aradiology subspecialty, such as neuroradiology or thoracic radiology(e.g., including subspecialties of chest and/or abdomen).

Act 4104 of flow diagram 4100 includes accessing a plurality ofcross-sectional medical images associated with a particular patient. Act4104 may be carried out using one or more components (e.g., I/O deviceinterface(s) 106, hardware processor(s) 108, image processing module(s)110, hardware storage device(s) 112, primary database 116, machinelearning module(s) 120, and/or others) of a computing system 100 orsimilar device. In some instances, one or more of the plurality ofcross-sectional medical images may comprise or be associated withmetadata indicating different radiology specialties that correspond todifferent subsets of the plurality of cross-sectional medical images.

In some instances, no (or incomplete) metadata is initially associatedwith the plurality of cross-sectional medical images. Thus, in someimplementations, Act 4104 may include prompting, at a user device, theuser to provide user input to associate one or more cross-sectionalmedical images within the plurality of cross-sectional medical imageswith one or more radiology specialties. Based on user input, metadatamay be stored for each of the one or more cross-sectional medicalimages, and the metadata may indicate the corresponding radiologyspecialty.

Act 4106 of flow diagram 4100 includes identifying a subset ofcross-sectional medical images from the plurality of cross-sectionalmedical images that correspond to the radiology specialty indicated bythe user profile. Act 4106 may be carried out using one or morecomponents (e.g., I/O device interface(s) 106, hardware processor(s)108, hardware storage device(s) 112, data processing module(s) 114,primary database 116, machine learning module(s) 120, and/or others) ofa computing system 100 or similar device. In some instances, identifyingthe subset of cross-sectional medical images that are associated withthe radiology specialty indicated by the user profile is based onmetadata associated with one or more of the cross-sectional medicalimages of the plurality of cross-sectional medical images (whetherestablished in accordance with act 4104 or otherwise).

Act 4108 of flow diagram 4100 includes presenting the subset ofcross-sectional medical images to the user in navigable form. Act 4108may be carried out using one or more components (e.g., I/O deviceinterface(s) 106, hardware processor(s) 108, image processing module(s)110, hardware storage device(s) 112, data processing module(s) 114,primary database 116, machine learning module(s) 120, and/or others) ofa computing system 100 or similar device.

Flow diagram 4100 may include acts associated with lesion analysisperformed using the subset of cross-sectional medical images of act4108.

For example, act 4110 of flow diagram 4100 includes rendering one ormore representations of a selection of the subset of cross-sectionalmedical images that include a marked lesion. Act 4110 may be carried outusing one or more components (e.g., I/O device interface(s) 106,hardware processor(s) 108, hardware storage device(s) 112, primarydatabase 116, and/or others) of a computing system 100 or similardevice. The one or more representations of the selection of the subsetof cross-sectional medical images that include a marked lesion may bedepicted, in some implementations, as elements of a list displayed to auser.

Act 4112 of flow diagram 4100 includes receiving user input selecting(i) a representation of the one or more representations of the selectionof the subset of cross-sectional medical images that include a markedlesion, or (ii) a control for navigating to a next cross-sectionalmedical image within the selection of the subset of cross-sectionalmedical images that include a marked lesion. Act 4112 may be carried outusing one or more components (e.g., I/O device interface(s) 106,hardware processor(s) 108, hardware storage device(s) 112, primarydatabase 116, and/or others) of a computing system 100 or similardevice. For example, where the one or more representations of theselection of the subset of cross-sectional medical images that include amarked lesion are depicted as elements of a list displayed to a user, auser may select one or more of the list elements. Similarly, a user maydirect user input toward a selectable button configured to navigate tothe cross-sectional image associated with a next or subsequent listelement.

Act 4114 of flow diagram 4100 includes navigating to and displaying (i)the cross-sectional medical image and the marked lesion that correspondto the representation of the one or more representations selected by theuser, or (ii) the next cross-sectional medical image and a next markedlesion in response to receiving user input selecting the control fornavigating to the next cross-sectional image within the selection of thesubset of cross-sectional medical images that include a marked lesion.Act 4114 may be carried out using one or more components (e.g., I/Odevice interface(s) 106, hardware processor(s) 108, hardware storagedevice(s) 112, primary database 116, and/or others) of a computingsystem 100 or similar device. In some instances, acts 4110, 4112, and4114 may facilitate guided presentations of lesions representedcross-sectional medical images to allow users to quickly review and/orassess analyzed lesions (e.g., as described with reference to FIGS.23-26).

In addition, or alternative, to acts 4110, 4112, and/or 4114 of flowdiagram 4100, acts 4116, 4118, 4120A, and/or 4120B of flow diagram 4100may be performed, which may facilitate lesion analysis (e.g., asdescribed with reference to FIGS. 2-22).

Act 4116 of flow diagram 4100 includes receiving user input thatconfigures the system to mark, segment, or label a lesion represented ina cross-sectional medical image of the subset of cross-sectional medicalimages. Act 4116 may be carried out using one or more components (e.g.,I/O device interface(s) 106, hardware processor(s) 108, image processingmodule(s) 110, hardware storage device(s) 112, data processing module(s)114, primary database 116, machine learning module(s) 120, and/orothers) of a computing system 100 or similar device. In someimplementations, the marking or segmentation of the lesion includesdetermining a predicted shape for the lesion (e.g., via the machinelearning module 120), and the labeling of the lesion includes providingthe predicted shape and/or other inputs to a machine learning module 120as inputs to determine anatomical location information for the lesion.

Act 4118 of flow diagram 4100 includes storing a subspecialty reportcomprising information associated with the marking, segmenting, orlabeling. Act 4102 may be carried out using one or more components(e.g., I/O device interface(s) 106, hardware processor(s) 108, imageprocessing module(s) 110, hardware storage device(s) 112, dataprocessing module(s) 114, primary database 116, machine learningmodule(s) 120, and/or others) of a computing system 100 or similardevice. A subspecialty report may comprise any information associatedwith an analyzed lesion, such as shape, major/minor axis, anatomicallocation, or even image data associated with the lesion. A subspecialtyreport may take the form of a formalized report ready for end use (e.g.,by a patient or an oncologist), or may simply comprise data stored in acomputer-readable manner.

Flow diagram 4100 includes a decision block 4120, which includesdetermining whether a subspecialty report has been stored for eachradiology specialty (in particular, a subspecialty report for eachradiology specialty that corresponds to a subset of cross-sectionalmedical images from the plurality of cross-sectional medical imagesassociated with the particular patient). In this regard, decision block4120 may comprise determining determine that marking, segmentation, orlabeling has been performed for each radiology specialty thatcorresponds to a subset of cross-sectional medical images from theplurality of cross-sectional medical images associated with theparticular patient.

Flow diagram 4100 illustrates that, in response to determining that asubspecialty report has been stored for each radiology specialtyaccording to decision block 4120, act 4120A may be performed. Act 4120Aincludes generating or updating a composite report comprising marking,segmentation, or labeling information for each cross-sectional medicalimage of the plurality of cross-sectional medical images associated withthe particular patient that includes marking, segmentation, or labeling(e.g., comprising lesion information from each subspecialty report). Act4120A may be carried out using one or more components (e.g., I/O deviceinterface(s) 106, hardware processor(s) 108, hardware storage device(s)112, data processing module(s) 114, primary database 116, and/or others)of a computing system 100 or similar device.

Flow diagram 4100 also illustrates that, in response to determining thata subspecialty report has not been stored for each radiology specialtyaccording to decision block 4120, act 4120B may be performed. Act 4120Bincludes refraining from generating or updating a composite reportcomprising marking, segmentation, or labeling information for eachcross-sectional medical image of the plurality of cross-sectionalmedical images associated with the particular patient that includesmarking, segmentation, or labeling.

Act 4202 of flow diagram 4200 of FIG. 42 includes identifying a userprofile associated with a user accessing a system. Act 4202 may becarried out using one or more components (e.g., I/O device interface(s)106, hardware processor(s) 108, hardware storage device(s) 112, dataprocessing module(s) 114, primary database 116, and/or others) of acomputing system 100 or similar device. In some instances, the userprofile indicates system interaction preferences for the user, and thesystem interaction preferences for the user may include an interactionpresentation.

The system interaction preferences include a different interactionpresentation for each of the plurality of controls. For example, theinteraction presentation may include a sound, an image, and/oranimation. The system interaction preferences may further include a userinterface theme that alters a rendering of at least some of theplurality of controls within the user interface (see act 4208).

Act 4204 of flow diagram 4200 includes accessing a plurality ofcross-sectional medical images. Act 4204 may be carried out using one ormore components (e.g., I/O device interface(s) 106, hardwareprocessor(s) 108, hardware storage device(s) 112, data processingmodule(s) 114, primary database 116, and/or others) of a computingsystem 100 or similar device. The plurality of cross-sectional medicalimages may comprise one or more of CT images, CTP images, PET images,SPECT images, MRI images, or ultrasound images, and/or others. Suchimages may be obtained by a radiologic device 104.

Act 4206 of flow diagram 4200 includes displaying the plurality ofcross-sectional medical images to the user in navigable form within auser interface. Act 4206 may be carried out using one or more components(e.g., I/O device interface(s) 106, hardware processor(s) 108, imageprocessing module(s) 110, hardware storage device(s) 112, dataprocessing module(s) 114, primary database 116, and/or others) of acomputing system 100 or similar device. The user interface may beoperated by a physician reviewer that is able to perform lesion analysison lesions represented in the plurality of cross-sectional medicalimages.

Act 4208 of flow diagram 4200 includes identifying a plurality ofcontrols within the user interface. Act 4208 may be carried out usingone or more components (e.g., I/O device interface(s) 106, hardwareprocessor(s) 108, hardware storage device(s) 112, primary database 116,and/or others) of a computing system 100 or similar device.

The plurality of controls may include, by way of non-limiting example,controls for: (i) selecting a position within a pixel regioncorresponding to a lesion represented in the cross-sectional medicalimages (e.g., the lesion may comprise a target lesion, non-targetlesion, or lesion that is neither a target nor a non-target lesion),(ii) tracing the pixel region associated with the lesion represented inthe cross-sectional medical images, (iii) selecting location informationfor the lesion, (iv) navigating through a guided presentation of asubset of cross-sectional medical images of the plurality ofcross-sectional medical images that include one or more marked lesions,(v) selecting a representation of a list entry associated with across-sectional medical image of the plurality of cross-sectionalmedical image that includes one or more marked lesions, (vi) accepting apredicted lesion shape or lesion location information generated by amachine learning module 120, (vii) rejecting a predicted lesion shape orlesion location information generated by a machine learning module,and/or (viii) triggering display of a report comprising informationassociated with one or more marked lesions present within the pluralityof cross-sectional medical images.

Act 4210 of flow diagram 4200 includes associating at least one theplurality of controls with the interaction presentation indicated in thesystem interaction preferences of the user profile. Act 4210 may becarried out using one or more components (e.g., I/O device interface(s)106, hardware processor(s) 108, hardware storage device(s) 112, dataprocessing module(s) 114, primary database 116, and/or others) of acomputing system 100 or similar device. Multiple controls may beassociated with the same or different interaction preferences.

Act 4212 of flow diagram 4200 includes detecting user input operatingthe at least one of the plurality of controls. Act 4212 may be carriedout using one or more components (e.g., I/O device interface(s) 106,hardware processor(s) 108, image processing module(s) 110, hardwarestorage device(s) 112, data processing module(s) 114, primary database116, machine learning module(s) 120, and/or others) of a computingsystem 100 or similar device. Such user input may be provided by aphysician reviewer performing lesion analysis using the plurality ofcross-sectional medical images.

Act 4214 of flow diagram 4200 includes presenting the interactionpresentation in response to detecting the user input operating the atleast one of the plurality of the controls. Act 4214 may be carried outusing one or more components (e.g., I/O device interface(s) 106,hardware processor(s) 108, image processing module(s) 110, hardwarestorage device(s) 112, data processing module(s) 114, primary database116, machine learning module(s) 120, and/or others) of a computingsystem 100 or similar device. In some instances, presenting interactionpresentations responsive to user input operating controls for lesionanalysis may at least partially ameliorate the mundanity associated withperforming lesion analysis, and such functionality may thereby improvethe alertness and engagement of physician reviewers, thereby increasingaccuracy and/or efficiency. In some implementations, when the at leastone of the plurality of controls is tracing the pixel region associatedwith the lesion, the interaction animation may persist while the usertraces the pixel region associated with the lesion.

Act 4302 of flow diagram 4300 of FIG. 43 includes accessing a first datastructure comprising one or more entries including location informationassociated with one or more lesions represented in one or morecross-sectional medical images from a first plurality of cross-sectionalmedical images of a patient. Act 4302 may be carried out using one ormore components (e.g., I/O device interface(s) 106, hardwareprocessor(s) 108, image processing module(s) 110, hardware storagedevice(s) 112, data processing module(s) 114, primary database 116,machine learning module(s) 120, and/or others) of a computing system 100or similar device. The one or more entries may comprise, for example, ashape associated with the one or more lesions and/or a major axis and/ora minor axis associated with the one or more lesions.

Act 4304 of flow diagram 4300 includes displaying representations ofeach of the one or more entries. Act 4304 may be carried out using oneor more components (e.g., I/O device interface(s) 106, hardwareprocessor(s) 108, hardware storage device(s) 112, data processingmodule(s) 114, primary database 116, and/or others) of a computingsystem 100 or similar device. In some implementations, therepresentations of each of the one or more entries may be displayed orpresented as elements of a list (e.g., as shown in FIGS. 28-32).

Act 4306 of flow diagram 4300 includes receiving user input selecting aparticular entry of the one or more entries. Act 4306 may be carried outusing one or more components (e.g., I/O device interface(s) 106,hardware processor(s) 108, hardware storage device(s) 112, and/orothers) of a computing system 100 or similar device. In some instances,the particular entry is selected via a user selection of therepresentation of the particular entry (e.g., selecting a list elementcorresponding to the particular entry). In some instances, theparticular entry is selected via a control for navigating to a nextentry, where the particular entry is the next entry.

Act 4308 of flow diagram 4300 includes presenting the cross-sectionalmedical image and the lesion represented therein associated with theparticular entry selected by the received user input. Act 4308 may becarried out using one or more components (e.g., I/O device interface(s)106, hardware processor(s) 108, hardware storage device(s) 112, primarydatabase 116, and/or others) of a computing system 100 or similardevice. In some instances, a lesion marker associated with theparticular entry is overlaid on the lesion represented in thecross-sectional medical image corresponding to the particular entry. Insome instances, a lesion marker may comprise an “X” mark, or majorand/or minor axes associated with the lesion represented in thecross-sectional medical image corresponding to the particular entry.

Act 4310 of flow diagram 4300 includes presenting a second plurality ofcross-sectional medical images in navigable form, wherein the secondplurality of cross-sectional medical images includes cross-sectionalimages of the patient captured at a timepoint different than a timepointassociated with the first plurality of cross-sectional medical images.

Act 4312 of flow diagram 4300 includes identify a predicted matchingcross-sectional medical image from the second plurality ofcross-sectional medical images that corresponds to the cross-sectionalmedical image associated with the particular entry (e.g., the particularentry of acts 4306 and 4308). Act 4312 may be carried out using one ormore components (e.g., I/O device interface(s) 106, hardwareprocessor(s) 108, image processing module(s) 110, hardware storagedevice(s) 112, data processing module(s) 114, primary database 116,machine learning module(s) 120, and/or others) of a computing system 100or similar device. In some instances, act 4312 is performed in responseto detecting the user input selecting the particular entry according toact 4306.

The predicted matching cross-sectional medical image may be identifiedusing various techniques and/or factors/inputs. For example, thepredicted matching cross-sectional medical image may be identified basedon a slice number associated with the particular entry. In someinstances, the predicted matching cross-sectional image may beidentified via image co-registration between the cross-sectional medicalimage associated with the particular entry and at least some of thesecond plurality of cross-sectional medical images. In someimplementations, the predicted matching cross-sectional medical image isidentified by directly comparing pixels between the cross-sectionalmedical image associated with the particular entry and at least some ofthe second plurality of cross-sectional medical images (e.g., via apixel patch comparison, image histogram analysis, intensity similarity,etc.).

Act 4314 of flow diagram 4300 includes presenting the predicted matchingcross-sectional medical image contemporaneously with presenting thecross-sectional medical image associated with the particular entryselected by the received user input. Act 4314 may be carried out usingone or more components (e.g., I/O device interface(s) 106, hardwareprocessor(s) 108, hardware storage device(s) 112, primary database 116,and/or others) of a computing system 100 or similar device. In someimplementations, the predicted matching cross-sectional image may bechanged or redefined based on user input received for navigating to adifferent cross-sectional medical image of the second plurality ofcross-sectional medical images.

Act 4316 of flow diagram 4300 includes receiving user input directed toa pixel region within a matching lesion in the predicted matchingcross-sectional medical image of the second plurality of cross-sectionalmedical images. Act 4316 may be carried out using one or more components(e.g., I/O device interface(s) 106, hardware processor(s) 108, imageprocessing module(s) 110, hardware storage device(s) 112, dataprocessing module(s) 114, primary database 116, machine learningmodule(s) 120, and/or others) of a computing system 100 or similardevice. The matching lesion may correspond to the lesion identified bythe lesion marker associated with the particular entry.

Act 4318 of flow diagram 4300 includes associating a matching lesionmarker with the pixel region within the matching lesion and storing thematching lesion marker in a matching entry in a second data structure.Act 4318 may be carried out using one or more components (e.g., I/Odevice interface(s) 106, hardware processor(s) 108, image processingmodule(s) 110, hardware storage device(s) 112, data processing module(s)114, primary database 116, machine learning module(s) 120, and/orothers) of a computing system 100 or similar device. The first andsecond data structures may comprise different parts of the same datastructure or may comprise separate data structures.

In some instances, act 4318 further includes copying the locationinformation for the lesion associated with the particular entry from theparticular entry into the matching entry (e.g., the lesion associatedwith the particular entry and the matching lesion may both compriserepresentations of the same physical lesion of the patient's body, suchthat location information for both the matching lesion and the lesionassociated with the particular entry may be the same)

Furthermore, act 4318 may also include storing segmentation informationassociated with the matching lesion into matching entry of the seconddatabase. The segmentation information may include a shape associatedwith the matching lesion and a major axis and/or a minor axis associatedwith the matching lesion. In some instances, the segmentationinformation is obtained at least in part by a machine learning module120.

Act 4320 of flow diagram 4300 includes generating a patient reportcomprising representations of information from at least the particularentry and the matching entry. Act 4320 may be carried out using one ormore components (e.g., I/O device interface(s) 106, hardwareprocessor(s) 108, image processing module(s) 110, hardware storagedevice(s) 112, data processing module(s) 114, primary database 116,machine learning module(s) 120, and/or others) of a computing system 100or similar device. The patient report may comprise any lesioninformation stored within the first data structure and/or the seconddata structure. In some instances, the patient report comprises an imageof the lesion associated with the particular entry and the matchinglesion.

Additional Details Concerning Computing Systems

As noted above, a computing system 100 may include and/or be used toperform any of the operations described herein. Computing system 100 maytake various different forms. For example, computing system 100 may beembodied as a tablet, a desktop, a laptop, a mobile device, a clouddevice, a head-mounted display, or a standalone device. Computing system100 may also be a distributed system that includes one or more connectedcomputing components/devices that are in communication with computingsystem 100.

Regarding the hardware processor(s) 108, it will be appreciated that thefunctionality described herein can be performed, at least in part, byone or more hardware logic components (e.g., the processor(s) 108). Thatis, any of the disclosed method acts and/or operations may be performedby the processor(s) 108. Illustrative types of hardware logiccomponents/processors that can be used include Field-Programmable GateArrays (“FPGA”), Program-Specific or Application-Specific IntegratedCircuits (“ASIC”), Program-Specific Standard Products (“ASSP”),System-On-A-Chip Systems (“SOC”), Complex Programmable Logic Devices(“CPLD”), Central Processing Units (“CPU”), Graphical Processing Units(“GPU”), or any other type of programmable hardware.

Hardware storage device(s) 112 may be physical system memory, which maybe volatile, non-volatile, or some combination of the two. The term“memory” may also be used herein to refer to non-volatile mass storagesuch as physical storage media. If computing system 100 is distributed,the processing, memory, and/or storage capability may be distributed aswell.

The disclosed embodiments may comprise or utilize a special-purpose orgeneral-purpose computer including computer hardware, such as, forexample, one or more processors (such as hardware processor(s) 108) andsystem memory (such as hardware storage device(s) 112), as discussed ingreater detail below. Embodiments also include physical and othercomputer-readable media for carrying or storing computer-executableinstructions and/or data structures. Such computer-readable media can beany available media that can be accessed by a general-purpose orspecial-purpose computer system. Computer-readable media that storecomputer-executable instructions in the form of data are “physicalcomputer storage media” or a “hardware storage device.”Computer-readable media that carry computer-executable instructions are“transmission media.” Thus, by way of example and not limitation, thecurrent embodiments can comprise at least two distinctly different kindsof computer-readable media: computer storage media and transmissionmedia.

Computer storage media (aka “hardware storage device”) arecomputer-readable hardware storage devices, such as RAM, ROM, EEPROM,CD-ROM, solid state drives (“SSD”) that are based on RAM, Flash memory,phase-change memory (“PCM”), or other types of memory, or other opticaldisk storage, magnetic disk storage or other magnetic storage devices,or any other medium that can be used to store desired program code meansin the form of computer-executable instructions, data, or datastructures and that can be accessed by a general-purpose orspecial-purpose computer.

Computing system 100 may also be connected (via a wired or wirelessconnection) to external sensors (e.g., one or more remote radiologicdevices 104) or devices via a network 128. For example, computing system100 can communicate with any number devices or cloud services to obtainor process data. In some cases, network 128 may itself be a cloudnetwork. Furthermore, computing system 100 may also be connected throughone or more wired or wireless networks 128 to remote/separate computersystems(s) that are configured to perform any of the processingdescribed with regard to computing system 100.

A “network,” like network 128, is defined as one or more data linksand/or data switches that enable the transport of electronic databetween computer systems, modules, and/or other electronic devices. Wheninformation is transferred, or provided, over a network (eitherhardwired, wireless, or a combination of hardwired and wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Computing system 100 will include one or more communicationchannels that are used to communicate with the network 128.Transmissions media include a network that can be used to carry data ordesired program code means in the form of computer-executableinstructions or in the form of data structures. Further, thesecomputer-executable instructions can be accessed by a general-purpose orspecial-purpose computer. Combinations of the above should also beincluded within the scope of computer-readable media.

Upon reaching various computer system components, program code means inthe form of computer-executable instructions or data structures can betransferred automatically from transmission media to computer storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface module (e.g., a network interface card or“NIC”) and then eventually transferred to computer system RAM and/or toless volatile computer storage media at a computer system. Thus, itshould be understood that computer storage media can be included incomputer system components that also (or even primarily) utilizetransmission media.

Computer-executable (or computer-interpretable) instructions comprise,for example, instructions that cause a general-purpose computer,special-purpose computer, or special-purpose processing device toperform a certain function or group of functions. Thecomputer-executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the embodiments may bepracticed in network computing environments with many types of computersystem configurations, including personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, and the like. The embodiments may alsobe practiced in distributed system environments where local and remotecomputer systems that are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network each perform tasks (e.g. cloud computing, cloudservices and the like). In a distributed system environment, programmodules may be located in both local and remote memory storage devices.

The concepts and features described herein may be embodied in otherspecific forms without departing from their spirit or descriptivecharacteristics. The described embodiments are to be considered in allrespects only as illustrative and not restrictive. The scope of thedisclosure is, therefore, indicated by the appended claims rather thanby the foregoing description. All changes which come within the meaningand range of equivalency of the claims are to be embraced within theirscope.

I claim:
 1. A system for analyzing lesions in cross-sectional medicalimages, comprising: one or more processors; and one or more hardwarestorage devices storing computer-executable instructions that areexecutable by the one or more processors to configure the system to:identify a user profile associated with a user accessing the system, theuser profile indicating a radiology specialty associated with the user;access a plurality of cross-sectional medical images associated with aparticular patient; identify a subset of cross-sectional medical imagesfrom the plurality of cross-sectional medical images that correspond tothe radiology specialty indicated by the user profile; present thesubset of cross-sectional medical images to the user in navigable form;receive user input that configures the system to mark, segment, or labela lesion represented in a cross-sectional medical image of the subset ofcross-sectional medical images; store a subspecialty report comprisinginformation associated with the marking, segmenting, or labeling;determine that marking, segmentation, or labeling has been performed foreach radiology specialty that corresponds to a subset of cross-sectionalmedical images from the plurality of cross-sectional medical imagesassociated with the particular patient; and in response to sodetermining, generate or update a composite report comprising marking,segmentation, or labeling information for each cross-sectional medicalimage of the plurality of cross-sectional medical images associated withthe particular patient that includes marking, segmentation, or labeling.2. The system of claim 1, wherein the radiology specialty is a radiologysubspecialty comprising any of: neuroradiology or body radiology,wherein body radiology comprises chest and/or abdomen subspecialties. 3.The system of claim 1, wherein identifying the subset of cross-sectionalmedical images that are associated with the radiology specialtyindicated by the user profile is based on metadata associated with oneor more of the cross-sectional medical images of the plurality ofcross-sectional medical images.
 4. The system of claim 1, wherein thecomputer-executable instructions are executable to configure the systemto: prompt, at a user device, the user to provide user input toassociate one or more cross-sectional medical images within theplurality of cross-sectional medical images with one or more radiologyspecialties; and store metadata for each of the one or morecross-sectional medical images, the metadata indicating thecorresponding radiology specialty.
 5. The system of claim 1, wherein thecomputer-executable instructions are executable to configure the systemto render one or more representations of a selection of the subset ofcross-sectional medical images that include a marked lesion.
 6. Thesystem of claim 5, wherein the computer-executable instructions areexecutable to configure the system to receive user input selecting: arepresentation of the one or more representations of the selection ofthe subset of cross-sectional medical images that include a markedlesion; or a control for navigating to a next cross-sectional medicalimage within the selection of the subset of cross-sectional medicalimages that include a marked lesion.
 7. The system of claim 6, whereinthe computer-executable instructions are executable to configure thesystem to navigate to and display: the cross-sectional medical image andthe marked lesion that correspond to the representation of the one ormore representations selected by the user; or the next cross-sectionalmedical image and a next marked lesion in response to receiving userinput selecting the control for navigating to the next cross-sectionalimage within the selection of the subset of cross-sectional medicalimages that include a marked lesion.
 8. The system of claim 1, whereinthe segmentation or labeling is performed at least in part utilizingmachine learning.
 9. The system of claim 1, wherein the subspecialtyreport includes a view of the lesion.
 10. The system of claim 1, whereinthe computer-executable instructions are executable to configure thesystem to: determine that marking, segmentation, or labeling has notbeen performed for each radiology specialty that corresponds to a subsetof cross-sectional medical images from the plurality of cross-sectionalmedical images associated with the particular patient; and in responseto so determining, refrain from generating or updating a compositereport comprising marking, segmentation, or labeling information foreach cross-sectional medical image of the plurality of cross-sectionalmedical images associated with the particular patient that includesmarking, segmentation, or labeling.
 11. A method for analyzing lesionsin cross-sectional medical images, comprising: identifying a userprofile associated with a user accessing a system, the user profileindicating a radiology specialty associated with the user; accessing aplurality of cross-sectional medical images associated with a particularpatient; identifying a subset of cross-sectional medical images from theplurality of cross-sectional medical images that correspond to theradiology specialty indicated by the user profile; and presenting thesubset of cross-sectional medical images to the user in navigable form;receiving user input that configures the system to mark, segment, orlabel a lesion represented in a cross-sectional medical image of thesubset of cross-sectional medical images; storing a subspecialty reportcomprising information associated with the marking, segmenting, orlabeling; determining that marking, segmentation, or labeling has beenperformed for each radiology specialty that corresponds to a subset ofcross-sectional medical images from the plurality of cross-sectionalmedical images associated with the particular patient; and in responseto so determining, generating or updating a composite report comprisingmarking, segmentation, or labeling information for each cross-sectionalmedical image of the plurality of cross-sectional medical imagesassociated with the particular patient that includes marking,segmentation, or labeling.
 12. The method of claim 11, wherein theradiology specialty is a radiology subspecialty comprising any of:neuroradiology or body radiology, wherein body radiology comprises chestand/or abdomen subspecialties.
 13. The method of claim 11, whereinidentifying the subset of cross-sectional medical images that areassociated with the radiology specialty indicated by the user profile isbased on metadata associated with one or more of the cross-sectionalmedical images of the plurality of cross-sectional medical images. 14.The method of claim 11, further comprising: prompting, at a user device,the user to provide user input to associate one or more cross-sectionalmedical images within the plurality of cross-sectional medical imageswith one or more radiology specialties; and storing metadata for each ofthe one or more cross-sectional medical images, the metadata indicatingthe corresponding radiology specialty.
 15. The method of claim 11,further comprising: determining that marking, segmentation, or labelinghas not been performed for each radiology specialty that corresponds toa subset of cross-sectional medical images from the plurality ofcross-sectional medical images associated with the particular patient;and in response to so determining, refraining from generating orupdating a composite report comprising marking, segmentation, orlabeling information for each cross-sectional medical image of theplurality of cross-sectional medical images associated with theparticular patient that includes marking, segmentation, or labeling. 16.A system for analyzing cross-sectional medical images, comprising: oneor more processors; and one or more hardware storage devices storingcomputer-executable instructions that are executable by the one or moreprocessors to configure the system to: access a plurality ofcross-sectional medical images associated with a particular patient;present the cross-sectional medical images or a subset thereof to theuser in navigable form; receive user input that configures the system tomark, segment, or label a cross-sectional medical image of thecross-sectional medical images; store a first subspecialty reportcomprising information associated with the marking, segmenting, orlabeling; determine that a second subspecialty report comprisingmarking, segmentation, or labeling of a cross-sectional image has beengenerated for at least one additional radiology specialty thatcorresponds to the cross-sectional medical images or a subset thereof;and in response to so determining, generate or update a composite reportcomprising the marking, segmentation, or labeling information associatedwith the first subspecialty report and the second subspecialty report.17. The system of claim 16, wherein the first subspecialty is aradiology subspecialty comprising any of: neuroradiology or bodyradiology, wherein body radiology comprises chest and/or abdomensubspecialties.
 18. The system of claim 16, further comprisingidentifying a subset of the cross-sectional medical images associatedwith the first subspecialty and presenting the subset to the user,wherein identifying the subset of cross-sectional medical images thatare associated with the first subspecialty is based on metadataassociated with one or more of the cross-sectional medical images of theplurality of cross-sectional medical images.
 19. The system of claim 16,wherein the computer-executable instructions are executable to configurethe system to: prompt, at a user device, the user to provide user inputto associate one or more cross-sectional medical images within theplurality of cross-sectional medical images with one or more radiologyspecialties; and store metadata for each of the one or morecross-sectional medical images, the metadata indicating thecorresponding radiology specialty.
 20. The system of claim 16, whereinthe computer-executable instructions are executable to configure thesystem to render one or more representations of a selection of a subsetof cross-sectional medical images that include a marked lesion.
 21. Thesystem of claim 16, wherein the first subspecialty report includes aview of a lesion.
 22. A system for analyzing cross-sectional medicalimages, comprising: one or more processors; and one or more hardwarestorage devices storing computer-executable instructions that areexecutable by the one or more processors to configure the system to:access a plurality of cross-sectional medical images associated with aparticular patient; identify a subset of cross-sectional medical imagesfrom the plurality of cross-sectional medical images that correspond toa first radiology specialty; present the subset of cross-sectionalmedical images to the user in navigable form; receive user input thatconfigures the system to mark, segment, or label a cross-sectionalmedical image of the subset of cross-sectional medical images; store afirst subspecialty report comprising information associated with themarking, segmenting, or labeling; determine that a second subspecialtyreport comprising marking, segmentation, or labeling of across-sectional image has been generated for at least one additionalradiology specialty that corresponds to a subset of cross-sectionalmedical images from the plurality of cross-sectional medical imagesassociated with the particular patient; and in response to sodetermining, generate or update a composite report comprising themarking, segmentation, or labeling information associated with the firstsubspecialty report and the second subspecialty report.